Artificial
IntelligenceIndex Report 2021

Artificial IntelligenceIndex Report 2021

INTRODUCTION TO THE
2021 AI INDEX REPORT
Welcome to the fourth edition of the AI Index Report!
This year we significantly expanded the amount of data
available in the report, worked with a broader set of
external organizations to calibrate our data, and deepened
our connections with Stanford’s Institute for HumanCentered Artificial Intelligence (HAI).
The AI Index Report tracks, collates, distills, and visualizes
data related to artificial intelligence. Its mission is toprovide unbiased, rigorously vetted, and globally sourced
data for policymakers, researchers, executives, journalists,
and the general public to develop intuitions about thecomplex field of AI. The report aims to be the world’s mostcredible and authoritative source for data and insights
about AI.

C OV I D A N D A I
The 2021 report shows the effects of COVID-19 on AI
development from multiple perspectives. The Technical
Performance chapter discusses how an AI startup usedmachine-learning-based techniques to accelerate COVIDrelated drug discovery during the pandemic, and our
Economy chapter suggests that AI hiring and private
investment were not significantly adversely influenced
by the pandemic, as both grew during 2020. If anything,
COVID-19 may have led to a higher number of people
participating in AI research conferences, as the pandemic
forced conferences to shift to virtual formats, which in turnled to significant spikes in attendance.

CHANGES FOR THIS EDITION
In 2020, we surveyed more than 140 readers fromgovernment, industry, and academia about what theyfound most useful about the report and what we shouldchange. The main suggested areas for improvement were:• Technical performance: We significantly expanded
this chapter in 2021, carrying out more of our own
analysis.

• Diversity and ethics data: We gathered more data
for this year’s report, although our investigation
surfaced several areas where the AI community
currently lacks good information.
• Country comparisons: Readers were generallyinterested in being able to use the AI Index for crosscountry comparisons. To support this, we:		• gathered more data to allow for comparison
among countries, especially relating toeconomics and bibliometrics; and		• included a thorough summary of the various AI
strategies adopted by different countries andhow they evolved over time.

P U B L I C DATA A N D TO O L S
The AI Index 2021 Report is supplemented by raw data
and an interactive tool. We invite each member of the AI
community to use the data and tool in a way most relevantto their work and interests.
• Raw data and charts: The public data and highresolution images of all the charts in the report areavailable on Google Drive.
• Global AI Vibrancy Tool: We revamped the Global AI
Vibrancy Tool this year, allowing for better interactive
visualization when comparing up to 26 countries
across 22 indicators. The updated tool provides
transparent evaluation of the relative position ofcountries based on users’ preference; identifies
relevant national indicators to guide policy priorities
at a country level; and shows local centers of AI
excellence for not just advanced economies but also
emerging markets.
• Issues in AI measurement: In fall 2020, we
published “Measurement in AI Policy: Opportunities
and Challenges,” a report that lays out a variety of
AI measurement issues discussed at a conference
hosted by the AI Index in fall 2019.

2

Artificial IntelligenceIndex Report 2021

Table of Contents
INTRODUCTION TO THE 2021 AI INDEX REPORT2

TOP 9 TAKEAWAYS

4

AI INDEX STEERING COMMITTEE & STAFF

5

HOW TO CITE THE REPORT6ACKNOWLEDGMENTS

7

REPORT HIGHLIGHTS

10

CHAPTER 1

Research and Development

14

CHAPTER 2

Technical Performance

41

CHAPTER 3

The Economy

80

CHAPTER 4

AI Education

107

CHAPTER 5

Ethical Challenges of AI Applications

125

CHAPTER 6

Diversity in AI

135

CHAPTER 7

AI Policy and National Strategies

151

APPENDIX		177

ACCESS THE PUBLIC DATA

3

Artificial IntelligenceIndex Report 2021

TOP 9 TAKEAWAYS

1

AI investment in drug design and discovery increased significantly: “Drugs, Cancer, Molecular,
Drug Discovery” received the greatest amount of private AI investment in 2020, with more than USD
13.8 billion, 4.5 times higher than 2019.

2

The industry shift continues: In 2019, 65% of graduating North American PhDs in AI went into
industry—up from 44.4% in 2010, highlighting the greater role industry has begun to play in AI
development.

3

Generative everything: AI systems can now compose text, audio, and images to a sufficiently highstandard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology.

45

AI has a diversity challenge: In 2019, 45% new U.S. resident AI PhD graduates were white—bycomparison, 2.4% were African American and 3.2% were Hispanic.

China overtakes the US in AI journal citations: After surpassing the United States in the total
number of journal publications several years ago, China now also leads in journal citations;however, the United States has consistently (and significantly) more AI conference papers (which are also
more heavily cited) than China over the last decade.

6

The majority of the US AI PhD grads are from abroad—and they’re staying in the US:
The percentage of international students among new AI PhDs in North America continued to rise in2019, to 64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States
and 8.6% have taken jobs outside the United States.

7

Surveillance technologies are fast, cheap, and increasingly ubiquitous: The technologies
necessary for large-scale surveillance are rapidly maturing, with techniques for image classification,
face recognition, video analysis, and voice identification all seeing significant progress in 2020.

8

AI ethics lacks benchmarks and consensus: Though a number of groups are producing a rangeof qualitative or normative outputs in the AI ethics domain, the field generally lacks benchmarksthat can be used to measure or assess the relationship between broader societal discussions about
technology development and the development of the technology itself. Furthermore, researchers andcivil society view AI ethics as more important than industrial organizations.

9

AI has gained the attention of the U.S. Congress: The 116th Congress is the most AI-focusedcongressional session in history with the number of mentions of AI in congressional record morethan triple that of the 115th Congress.

4Artificial IntelligenceIndex Report 2021

AI Index Steering Committee
Co-Directors
Jack Clark
OECD, GPAI

Raymond Perrault
SRI International

Members
Erik Brynjolfsson
Stanford University

James Manyika
McKinsey Global InstituteJohn Etchemendy
Stanford University

Juan Carlos Niebles
Stanford University

Deep Ganguli
Stanford University

Michael Sellitto
Stanford University

Barbara Grosz
Harvard University

Yoav Shoham (Founding Director)
Stanford University, AI21 Labs

Terah Lyons
Partnership on AI

AI Index Staff
Research Manager and Editor in Chief
Daniel Zhang
Stanford University

Program Manager
Saurabh Mishra
Stanford University

5Artificial IntelligenceIndex Report 2021

How to Cite This Report
Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara
Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham,Jack Clark, and Raymond Perrault, “The AI Index 2021 Annual Report,” AI Index Steering
Committee, Human-Centered AI Institute, Stanford University, Stanford, CA, March 2021.
The AI Index 2021 Annual Report by Stanford University is licensed under
Attribution-NoDerivatives 4.0 International. To view a copy of this license,
visit http://creativecommons.org/licenses/by-nd/4.0/.

The AI Index is an independent initiative at Stanford University’s
Human-Centered Artificial Intelligence Institute (HAI).

The AI Index was conceived within the One Hundred Year Study on AI (AI100).

We thank our supporting partners

We welcome feedback and new ideas for next year.
Contact us at AI-Index-Report@stanford.edu.

6Artificial IntelligenceIndex Report 2021

Acknowledgments
We appreciate the following organizations and individuals who provided data, analysis,
advice, and expert commentary for inclusion in the AI Index 2021 Report:

Organizations
arXiv
Jim Entwood, Paul Ginsparg,
Joe Halpern, Eleonora Presani

LinkedIn
Guy Berger, Mar Carpanelli, Di Mo,
Virginia Ramsey

AI Ethics Lab
Cansu Canca, Yasemin Usta

Liquidnet
Jeffrey Banner, Steven Nichols

Black in AI
Rediet Abebe, Hassan Kane

McKinsey Global Institute
Brittany Presten

Bloomberg Government
Chris Cornillie

Microsoft Academic Graph
Iris Shen, Kuansan Wang

Burning Glass Technologies
Layla O’Kane, Bledi Taska, Zhou Zhou

National Institute of Standards andTechnology
Patrick Grother

Computing Research Association
Andrew Bernat, Susan Davidson
Elsevier
Clive Bastin, Jörg Hellwig,
Sarah Huggett, Mark Siebert
Intento
Grigory Sapunov, Konstantin Savenkov
International Federation of Robotics
Susanne Bieller, Jeff Burnstein
Joint Research Center, European
Commission
Giuditta De Prato, Montserrat López
Cobo, Riccardo Righi

Nesta
Joel Klinger, Juan Mateos-Garcia,
Kostas Stathoulopoulos
NetBase Quid
Zen Ahmed, Scott Cohen, Julie Kim
PostEraAaron Morris
Queer in AI
Raphael Gontijo Lopes
State of AI ReportNathan Benaich, Ian Hogarth
Women in Machine Learning
Sarah Tan, Jane Wang
7

Artificial IntelligenceIndex Report 2021

Individuals
ActivityNet
Fabian Caba (Adobe Research); Bernard
Ghanem (King Abdullah University of
Science and Technology); Cees Snoek(University of Amsterdam)
AI Brain Drain and Faculty Departure
Michael Gofman (University of Rochester); Zhao Jin (Cheung Kong Graduate
School of Business)
Automated Theorem Proving
Geoff Sutcliffe (University of Miami);
Christian Suttner (Connion GmbH)
Boolean Satisfiability Problem
Lars Kotthoff (University of Wyoming)
Corporate Representation at
AI Research Conferences
Nuruddin Ahmed (Ivey Business School,
Western University); Muntasir Wahed(Virginia Tech)

Ethics at AI Conferences
Pedro Avelar, Luis Lamb, Marcelo
Prates (Federal University of Rio
Grande do Sul)
ImageNet
Lucas Beyer, Alexey Dosovitskiy,
Neil Houlsby (Google)
MLPerf/DAWNBench
Cody Coleman (Stanford University),Peter Mattson (Google)
Molecular Synthesis
Philippe Schwaller (IBMResearch–Europe)
Visual Question Answering
Dhruv Batra, Devi Parikh (Georgia
Tech/FAIR); Ayush Shrivastava(Georgia Tech)
You Only Look Once
Xiang Long (Baidu)

Conference Attendance
Maria Gini, Gita Sukthankar (AAMAS);
Carol Hamilton (AAAI); Dan Jurafsky(ACL); Walter Scheirer, Ramin Zabih(CVPR); Jörg Hoffmann, Erez Karpas(ICAPS); Paul Oh (IROS); Pavlos Peppas,
Michael Thielscher (KR)8

Artificial IntelligenceIndex Report 2021

Advice and Expert Commentary
Alexey Bochkovskiy; Baidu’s PaddlePaddle Computer Vision Team; Chenggang Xu (Cheung Kong
Graduate School of Business); Mohammed AlQuraishi (Columbia University); Evan Schnidman(EAS Innovation); Fanghzhen Lin (Hong Kong University of Science and Technology); David Kanter(MLCommons); Sam Bowman (New York University); Maneesh Agrawala, Jeannette Bohg, Emma
Brunskill, Chelsea Finn, Aditya Grover, Tatsunori Hashimoto, Dan Jurafsky, Percy Liang, Sharon
Zhou (Stanford University); Vamsi Sistla (University of California, Berkeley); Simon King (Universityof Edinburgh); Ivan Goncharov (Weights & Biases)

Graduate Researchers
Ankita Banerjea, Yu-chi Tsao (Stanford University)

Report and Website
Michi Turner (report graphic design and cover art); Nancy King (report editor); Michael Taylor(report data visualization); Kevin Litman-Navarro (Global AI Vibrancy Tool design and development);
Travis Tyler (AI Index website design); Digital Avenues (AI Index website development)

9Artificial IntelligenceIndex Report 2021

REPORT HIGHLIGHTSC H A P T E R 1 : R E S E A R C H & D E V E LO P M E N T• The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher percentage growththan from 2018 to 2019 (19.6%).
• In every major country and region, the highest proportion of peer-reviewed AI papers comes from academic
institutions. But the second most important originators are different: In the United States, corporateaffiliated research represents 19.2% of the total publications, whereas government is the second mostimportant in China (15.6%) and the European Union (17.2%).
• In 2020, and for the first time, China surpassed the United States in the share of AI journal citations in theworld, having briefly overtaken the United States in the overall number of AI journal publications in 2004 andthen retaken the lead in 2017. However, the United States has consistently (and significantly) more cited AI
conference papers than China over the last decade.
• In response to COVID-19, most major AI conferences took place virtually and registered a significant increase
in attendance as a result. The number of attendees across nine conferences almost doubled in 2020.
• In just the last six years, the number of AI-related publications on arXiv grew by more than sixfold, from 5,478
in 2015 to 34,736 in 2020.
• AI publications represented 3.8% of all peer-reviewed scientific publications worldwide in 2019, up from
1.3% in 2011.

C H A P T E R 2: T E C H N I CA L P E R F O R M A N C E• Generative everything: AI systems can now compose text, audio, and images to a sufficiently high standard
that humans have a hard time telling the difference between synthetic and non-synthetic outputs for someconstrained applications of the technology. That promises to generate a tremendous range of downstream
applications of AI for both socially useful and less useful purposes. It is also causing researchers to invest intechnologies for detecting generative models; the DeepFake Detection Challenge data indicates how wellcomputers can distinguish between different outputs.
• The industrialization of computer vision: Computer vision has seen immense progress in the past decade,primarily due to the use of machine learning techniques (specifically deep learning). New data shows thatcomputer vision is industrializing: Performance is starting to flatten on some of the largest benchmarks,
suggesting that the community needs to develop and agree on harder ones that further test performance.
Meanwhile, companies are investing increasingly large amounts of computational resources to train
computer vision systems at a faster rate than ever before. Meanwhile, technologies for use in deployedsystems—like object-detection frameworks for analysis of still frames from videos—are maturing rapidly,
indicating further AI deployment.

10Artificial IntelligenceIndex Report 2021

• Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has yielded AI
systems with significantly improved language capabilities that have started to have a meaningful economic impact
on the world. Google and Microsoft have both deployed the BERT language model into their search engines, whileother large language models have been developed by companies ranging from Microsoft to OpenAI. Progress in
NLP has been so swift that technical advances have started to outpace the benchmarks to test for them. This canbe seen in the rapid emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation
suite developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
• New analyses on reasoning: Most measures of technical problems show for each time point the performanceof the best system at that time on a fixed benchmark. New analyses developed for the AI Index offer metrics thatallow for an evolving benchmark, and for the attribution to individual systems of credit for a share of the overall
performance of a group of systems over time. These are applied to two symbolic reasoning problems, AutomatedTheorem Proving and Satisfiability of Boolean formulas.
• Machine learning is changing the game in healthcare and biology: The landscape of the healthcare andbiology industries has evolved substantially with the adoption of machine learning. DeepMind’s AlphaFold
applied deep learning technique to make a significant breakthrough in the decades-long biology challenge ofprotein folding. Scientists use ML models to learn representations of chemical molecules for more effective
chemical synthesis planning. PostEra, an AI startup used ML-based techniques to accelerate COVID-related drug
discovery during the pandemic.

C H A P T E R 3: T H E E C O N O M Y• “Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI investment in 2020, withmore than USD 13.8 billion, 4.5 times higher than 2019.
• Brazil, India, Canada, Singapore, and South Africa are the countries with the highest growth in AI hiring from2016 to 2020. Despite the COVID-19 pandemic, the AI hiring continued to grow across sample countries in 2020.
• More private investment in AI is being funneled into fewer startups. Despite the pandemic, 2020 saw a 9.3%
increase in the amount of private AI investment from 2019—a higher percentage increase than from 2018 to 2019(5.7%), though the number of newly funded companies decreased for the third year in a row.
• Despite growing calls to address ethical concerns associated with using AI, efforts to address these concerns
in the industry are limited, according to a McKinsey survey. For example, issues such as equity and fairness
in AI continue to receive comparatively little attention from companies. Moreover, fewer companies in 2020
view personal or individual privacy risks as relevant, compared with in 2019, and there was no change in thepercentage of respondents whose companies are taking steps to mitigate these particular risks.
• Despite the economic downturn caused by the pandemic, half the respondents in a McKinsey survey said thatthe coronavirus had no effect on their investment in AI, while 27% actually reported increasing their investment.
Less than a fourth of businesses decreased their investment in AI.
• T he United States recorded a decrease in its share of AI job postings from 2019 to 2020—the first drop in
six years. The total number of AI jobs posted in the United States also decreased by 8.2%, from 325,724
in 2019 to 300,999 in 2020.

11

Artificial IntelligenceIndex Report 2021

C H A P T E R 4 : A I E D U CAT I O N• An AI Index survey conducted in 2020 suggests that the world’s top universities have increased their investment
in AI education over the past four years. The number of courses that teach students the skills necessary to build
or deploy a practical AI model on the undergraduate and graduate levels has increased by 102.9% and 41.7%,
respectively, in the last four academic years.
• More AI PhD graduates in North America chose to work in industry in the past 10 years, while fewer opted for jobsin academia, according to an annual survey from the Computing Research Association (CRA). The share of new
AI PhDs who chose industry jobs increased by 48% in the past decade, from 44.4% in 2010 to 65.7% in 2019. Bycontrast, the share of new AI PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019.
• In the last 10 years, AI-related PhDs have gone from 14.2% of the total of CS PhDs granted in the United States,
to around 23% as of 2019, according to the CRA survey. At the same time, other previously popular CS PhDs havedeclined in popularity, including networking, software engineering, and programming languages. Compilers all
saw a reduction in PhDs granted relative to 2010, while AI and Robotics/Vision specializations saw a substantial
increase.
• After a two-year increase, the number of AI faculty departures from universities to industry jobs in North America
dropped from 42 in 2018 to 33 in 2019 (28 of these are tenured faculty and five are untenured). Carnegie Mellon
University had the largest number of AI faculty departures between 2004 and 2019 (16), followed by the Georgia
Institute of Technology (14) and University of Washington (12).
• The percentage of international students among new AI PhDs in North America continued to rise in 2019, to
64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States and 8.6% havetaken jobs outside the United States.
• In the European Union, the vast majority of specialized AI academic offerings are taught at the master’s level;robotics and automation is by far the most frequently taught course in the specialized bachelor’s and master’s
programs, while machine learning (ML) dominates in the specialized short courses.

C H A P T E R 5: E T H I CA L C H A L L E N G E S O F A I A P P L I CAT I O N S• The number of papers with ethics-related keywords in titles submitted to AI conferences has grown since 2015,
though the average number of paper titles matching ethics-related keywords at major AI conferences remains
low over the years.
• The five news topics that got the most attention in 2020 related to the ethical use of AI were the release of theEuropean Commission’s white paper on AI, Google’s dismissal of ethics researcher Timnit Gebru, the AI ethics
committee formed by the United Nations, the Vatican’s AI ethics plan, and IBM’s exiting the facial-recognitionbusinesses.

12Artificial IntelligenceIndex Report 2021

CHAPTER 6: DIVERSITY IN AI• The percentages of female AI PhD graduates and tenure-track computer science (CS) faculty have remained low
for more than a decade. Female graduates of AI PhD programs in North America have accounted for less than
18% of all PhD graduates on average, according to an annual survey from the Computing Research Association(CRA). An AI Index survey suggests that female faculty make up just 16% of all tenure-track CS faculty at several
universities around the world.
• The CRA survey suggests that in 2019, among new U.S. resident AI PhD graduates, 45% were white, while 22.4%
were Asian, 3.2% were Hispanic, and 2.4% were African American.
• The percentage of white (non-Hispanic) new computing PhDs has changed little over the last 10 years,
accounting for 62.7% on average. The share of Black or African American (non-Hispanic) and Hispanic computing
PhDs in the same period is significantly lower, with an average of 3.1% and 3.3%, respectively.
• The participation in Black in AI workshops, which are co-located with the Conference on Neural Information
Processing Systems (NeurIPS), has grown significantly in recent years. The numbers of attendees and submitted
papers in 2019 are 2.6 times higher than in 2017, while the number of accepted papers is 2.1 times higher.
• In a membership survey by Queer in AI in 2020, almost half the respondents said they view the lack ofinclusiveness in the field as an obstacle they have faced in becoming a practitioner in the AI/ML field. More than
40% of members surveyed said they have experienced discrimination or harassment at work or school.

C H A P T E R 7: A I P O L I CY A N D N AT I O N A L S T R AT E G I E S• Since Canada published the world’s first national AI strategy in 2017, more than 30 other countries and regions
have published similar documents as of December 2020.
• The launch of the Global Partnership on AI (GPAI) and Organisation for Economic Co-operation and Development(OECD) AI Policy Observatory and Network of Experts on AI in 2020 promoted intergovernmental efforts to work
together to support the development of AI for all.
• In the United States, the 116th Congress was the most AI-focused congressional session in history. The numberof mentions of AI by this Congress in legislation, committee reports, and Congressional Research Service (CRS)
reports is more than triple that of the 115th Congress.

13Artificial IntelligenceIndex Report 2021

CHAPTER 1:

Research &Development

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

14

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

CHAPTER 1:

Chapter Preview
Overview

16

Chapter Highlights

17

1.1 PUBLICATIONS

18

Peer-Reviewed AI Publications

AI Patents

31

Overview

31

arXiv Publications

32

Overview

32

18

By Region

32

Overview

18

By Geographic Area

33

By Region

18

By Field of Study34

By Geographic Area

20

By Institutional Affiliation

21

Academic-Corporate Collaboration

23

1.2 CONFERENCES

36

25

Conference Attendance

36

Overview

25

By Region

26

Highlight: Corporate Representation
at AI Research Conferences

38

By Geographic Area

27

Citation27

1.3 AI OPEN-SOURCE SOFTWARELIBRARIES

39

GitHub Stars

39

AI Journal Publications

AI Conference Publications

28

Overview

28

By Region

29

By Geographic Area

30

Citation

30

Highlight: Deep Learning Papers on arXiv 35

ACCESS THE PUBLIC DATACHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

15

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OV E R V I E W

Overview
The report opens with an overview of the research and development (R&D)
efforts in artificial intelligence (AI) because R&D is fundamental to AI
progress. Since the technology first captured the imagination of computerscientists and mathematicians in the 1950s, AI has grown into a major
research discipline with significant commercial applications. The numberof AI publications has increased dramatically in the past 20 years. The riseof AI conferences and preprint archives has expanded the dissemination ofresearch and scholarly communications. Major powers, including China, theEuropean Union, and the United States, are racing to invest in AI research.
The R&D chapter aims to capture the progress in this increasingly complex
and competitive field.
This chapter begins by examining AI publications—from peer-reviewed
journal articles to conference papers and patents, including the citation
impact of each, using data from the Elsevier/Scopus and Microsoft
Academic Graph (MAG) databases, as well as data from the arXiv paper
preprint repository and Nesta. It examines contributions to AI R&Dfrom major AI entities and geographic regions and considers how thosecontributions are shaping the field. The second and third sections discuss
R&D activities at major AI conferences and on GitHub.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

16

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher
percentage growth than from 2018 to 2019 (19.6%).
• In every major country and region, the highest proportion of peer-reviewed AI papers
comes from academic institutions. But the second most important originators aredifferent: In the United States, corporate-affiliated research represents 19.2% of the total
publications, whereas government is the second most important in China (15.6%) and theEuropean Union (17.2%).
• In 2020, and for the first time, China surpassed the United States in the share of AI
journal citations in the world, having briefly overtaken the United States in the overall
number of AI journal publications in 2004 and then retaken the lead in 2017. However,the United States has consistently (and significantly) more cited AI conference papers
than China over the last decade.
• In response to COVID-19, most major AI conferences took place virtually and registereda significant increase in attendance as a result. The number of attendees across nine
conferences almost doubled in 2020.
• In just the last six years, the number of AI-related publications on arXiv grew by morethan sixfold, from 5,478 in 2015 to 34,736 in 2020.
• AI publications represented 3.8% of all peer-reviewed scientific publications worldwide
in 2019, up from 1.3% in 2011.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

17

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

AI publications include peer-reviewed publications, journal articles, conference papers, and patents. To track trends among thesepublications and to assess the state of AI R&D activities around the world, the following datasets were used: the Elsevier/Scopus
database for peer-reviewed publications; the Microsoft Academic Graph (MAG) database for all journals, conference papers, andpatent publications; and arXiv and Nesta data for electronic preprints.

1.1 PUBLICATIONSP E E R - R E V I E W E D A I P U B L I CAT I O N SThis section presents data from the Scopus database by
Elsevier. Scopus contains 70 million peer-reviewed research
items curated from more than 5,000 international publishers.
The 2019 version of the data shown below is derived froman entirely new set of publications, so figures of all peerreviewed AI publications differ from those in previous years’
AI Index reports. Due to changes in the methodology forindexing publications, the accuracy of the dataset increased
from 80% to 84% (see the Appendix for more details).

Overview
Figure 1.1.1a shows the number of peer-reviewed AI
publications, and Figure 1.1.1b shows the share of thoseamong all peer-reviewed publications in the world. The
total number of publications grew by nearly 12 times
between 2000 and 2019. Over the same period, thepercentage of peer-reviewed publications increased from
0.82% of all publications in 2000 to 3.8% in 2019.

By Region1
Among the total number of peer-reviewed AI publicationsin the world, East Asia & Pacific has held the largest sharesince 2004, followed by Europe & Central Asia, and North
America (Figure 1.1.2). Between 2009 and 2019, South Asiaand sub-Saharan Africa experienced the highest growthin terms of the number of peer-reviewed AI publications,increasing by eight- and sevenfold, respectively.

NUMBER of PEER-REVIEWED AI PUBLICATIONS, 2000-19

120

100

80

60

40

20

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0
2000

Number of Peer-Reviewed AI Publications (in Thousands)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Figure 1.1.1a
1 Regions in this chapter are classified according to the World Bank analytical grouping.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

18

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

PEER-REVIEWED AI PUBLICATIONS (% of TOTAL), 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index ReportPeer-Reviewed AI Publications (% of Total)

4%

3.8%

3%

2%

1%

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.1b

PEER-REVIEWED AI PUBLICATIONS (% of TOTAL) by REGION, 2000-19
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index ReportPeer-Reviewed AI Publications (% of Total)

40%
36.9% East Asia & Pacific30%
25.1% Europe & Central Asia

20%
17.0% North America

10%8.8% South Asia
5.5% Middle East & North Africa
2.7% Latin America & Caribbean

2023

2022

2021

2020

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

2019

0.7% Sub-Saharan Africa

0%Figure 1.1.2

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

19

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
To compare the activity among the world’s major AI
players, this section shows trends of peer-reviewed AI
publications coming out of China, the European Union,
and the United States. As of 2019, China led in the shareof peer-reviewed AI publications in the world, after

overtaking the European Union in 2017 (Figure 1.1.3).
It published 3.5 times more peer-reviewed AI papers
in 2019 than it did in 2014—while the European Union
published just 2 times more papers and the United States
2.75 times more over the same period.

PEER-REVIEWED AI PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-19

Peer-Reviewed AI Publications (% of World Total)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report25%
22.4% China

20%

16.4% EU

15%
14.6% US

10%

5%

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.3

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

20CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Institutional Affiliation
The following charts show the number of peer-reviewed
AI publications affiliated with corporate, government,
medical, and other institutions in China (Figure 1.1.4a),the European Union (Figure 1.1.4b), and the United States(Figure 1.1.4c).2 In 2019, roughly 95.4% of overall peerreviewed AI publications in China were affiliated with theacademic field, compared with 81.9% in the European
Union and 89.6% in the United States. Those affiliation
categories are not mutually exclusive, as some authors

could be affiliated with more than one type of institution.
The data suggests that, excluding academia, government
institutions—more than those in other categories—
consistently contribute the highest percentage of peerreviewed AI publications in both China and the European
Union (15.6% and 17.2 %, respectively, in 2019), whilein the United States, the highest portion is corporateaffiliated (19.2%).

NUMBER of PEER-REVIEWED AI PUBLICATIONS in CHINA by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

4,000

3,000

2,000

1,675 Corporate

1,000
382 Medical
14 Other

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

2019

Number of Peer-Reviewed AI Publications

4,352 Government

Figure 1.1.4a

2 Across all three geographic areas, the number of papers affiliated with academia exceeds that of government-, corporate-, and medical-affiliated ones; therefore, the academia affiliation is not shown,as it would distort the graphs.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

21

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

NUMBER of PEER-REVIEWED AI PUBLICATIONS in the EUROPEAN UNION by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Number of Peer-Reviewed AI Publications

3,523 Government

3,000

2,000
1,594 Corporate

1,000
508 Medical
187 Other

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.4b

NUMBER of PEER-REVIEWED AI PUBLICATIONS in the UNITED STATES by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Number of Peer-Reviewed AI Publications

3,513 Corporate

3,000

2,277 Gov.

2,000

1,000

718 Medical

120 Other

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.4c

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

22

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

Academic-Corporate Collaboration
Since the 1980s, the R&D collaboration between
academia and industry in the United States has grownin importance and popularity, made visible by the
proliferation of industry-university research centers aswell as corporate contributions to university research.
Figure 1.1.5 shows that between 2015 and 2019, theUnited States produced the highest number of hybrid
academic-corporate, co-authored, peer-reviewed AI
publications—more than double the amount in theEuropean Union, which comes in second, followed by
China in third place.

NUMBER of ACADEMIC-CORPORATE PEER-REVIEWED AI PUBLICATIONS by GEOGRAPHIC AREA, 2015-19 (SUM)
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

United States
European Union
China
United Kingdom
Germany
Japan
FranceCanadaSouth Korea
Netherlands
Switzerland
India
Hong Kong
Spain
Italy
0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

Number of Peer-Reviewed AI Publications
Figure 1.1.5

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

23

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

according to the world average. For example, an FWCI of
0.75 means 25% fewer citations than the world average.

To assess how academic-corporate collaborations
impact the Field-Weighted Citation Impact (FWCI) of
AI publications from different geographic regions,
see Figure 1.1.6. FWCI measures how the number ofcitations received by publications compares with theaverage number of citations received by other similar
publications in the same year, discipline, and format(book, article, conference paper, etc.). A value of 1.0
represents the world average. More than or less than 1means publications are cited more or less than expected,

The chart shows the FWCI for all peer-reviewed AI
publications on the y-axis and the total number (on a log
scale) of academic-corporate co-authored publications
on the x-axis. To increase the signal-to-noise ratio of the
FWCI metric, only countries that have more than 1,000
peer-reviewed AI publications in 2020 are included.

PEER-REVIEWED AI PUBLICATIONS' FIELD-WEIGHTED CITATION IMPACT and NUMBER of ACADEMIC-CORPORATE
PEER-REVIEWED AI PUBLICATIONS, 2019
Peer-Reviewed AI Publications' Field-Weighted Citation Impact (FWCI)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report3

Singapore

Hong Kong

Australia

Switzerland

Netherlands
2

Iran

CanadaMalaysia

Italy
Spain

GermanySouth Korea
European Union

FranceTurkeyRussiaIndonesia

China

Japan

Taiwan1

BrazilUnited States

United Kingdom

India

0
5

10

20

50

100

200

500

1,000

2,000

Number of Academic-Corporate Peer-Reviewed AI Publications (Log Scale)Figure 1.1.6

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

24

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I J O U R N A L P U B L I CAT I O N S

Overview

The next three sections chart the trends in the publicationof AI journals, conference publications, and patents, aswell as their respective citations that provide a signal
for R&D impact, based on data from Microsoft Academic
Graph. MAG3 is a knowledge graph consisting of more than
225 million publications (at the end of November 2019).

Overall, the number of AI journal publications in 2020 is 5.4
times higher than it was in 2000 (Figure 1.1.7a). In 2020, thenumber of AI journal publications increased by 34.5% from2019—a much higher percentage growth than from 2018 to2019 (19.6%). Similarly, the share of AI journal publications
among all publications in the world has jumped by 0.4
percentage points in 2020, higher than the average of 0.03
percentage points in the past five years (Figure 1.1.7b).

NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Journal Publications (in Thousands)

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.7a

AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20

3%

2.2%

2%

1%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Journal Publications (% of All Publications)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.7b
3 See “An Overview of Microsoft Academic Service (MAS) and Applications” and “A Review of Microsoft Academic Services for Science of Science Studies” for more details.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

25

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Region
Figure 1.1.8 shows the share of AI journals—the dominantpublication entity in terms of numbers in the MAG
database—by region between 2000 and 2020. East Asia
& Pacific, Europe & Central Asia, and North America are
responsible for the majority of AI journal publications in
the past 21 years, while the lead position among the three

regions changes over time. In 2020, East Asia & Pacific held
the highest share (26.7%), followed by Europe & Central
Asia (13.3%) and North America (14.0%). Additionally,in the last 10 years, South Asia, and Middle East & North
Africa saw the most significant growth, as the number of
AI journal publications in those two regions grew six- andfourfold, respectively.

AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report30%
26.7%, East Asia & Pacific20%
14.0%, North America
13.3%, Europe & Central Asia

10%4.9%, South Asia

2024

2023

2022

2021

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%

2020

3.1%, Middle East & North Africa
1.3%, Latin America & Caribbean
0.3%, Sub-Saharan Africa

2000

AI Journal Publications (% of World Total)

40%

Figure 1.1.8

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

26

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
Figure 1.1.9 shows that among the three major AI powers, China has had the largest share of AI journal publications in theworld since 2017, with 18.0% in 2020, followed by the United States (12.3%) and the European Union (8.6%).
AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
AI Journal Publications (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%

18.0%, China

15%
12.3%, US

10%8.6%, EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.9

Citation

In terms of the highest share of AI journal citations, Figure 1.1.10 shows that China (20.7%) overtook the United States(19.8%) in 2020 for the first time, while the European Union continued to lose overall share.
AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Citations (% of World Total)

40%

30%

20.7%
China

20%

19.8%
US

10%

11.0%
EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.10

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

27

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I C O N F E R E N C E P U B L I CAT I O N S

Overview

Between 2000 and 2019, the number of AI conference publications increased fourfold, although the growth flattenedout in the past ten years, with the number of publications in 2019 just 1.09 times higher than the number in 2010.4
NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of Publications (in Thousands)

50

40

30

20

10

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.11a

AI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of All Publications)

25%
20.2%

20%

15%

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.11b
4 Note that conference data in 2020 on the MAG system is not yet complete. See the Appendix for details.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

28

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Region
Asia & Pacific took the lead starting in 2004, accounting
for more than 27% in 2020. North America overtook
Europe & Central Asia to claim second place in 2018,accounting for 20.1%, followed by 21.7% in 2020.

Figure 1.1.12 shows that, similar to the trends in AI
journal publication, East Asia & Pacific, Europe & Central
Asia, and North America are the world’s dominantsources for AI conference publications. Specifically, East

AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report30%
27.3%, East Asia & Pacific21.7%, North America

20%

18.6%, Europe & Central Asia

10%

2026

2025

2024

2023

2022

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

1999

2000

0%

2021

5.1%, South Asia
2.2%, Middle East & North Africa
1.7%, Latin America & Caribbean
0.3%, Sub-Saharan Africa

2020

AI Conference Publications (% of World Total)

40%

Figure 1.1.12

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

29

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
is almost nine times higher than it was in 2000. The
share of conference publications for the European Union
peaked in 2011 and continues to decline.

China overtook the United States in the share of AI
conference publications in the world in 2019 (Figure
1.1.13). Its share has grown significantly since 2000.
China’s percentage of AI conference publications in 2019

AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20

AI Conference Publications (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%

19.4% US15%

15.2% China
12.8% EU

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.13

Citation

21 years. The United States tops the list with 40.1% of
overall citations in 2020, followed by China (11.8%) andthe European Union (10.9%).

With respect to citations of AI conference publications,Figure 1.1.14 shows that the United States has held adominant lead among the major powers over the past

AI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
40.1% US

40%

30%

20%
11.8% China

10%

10.9% EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Conference Citations (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.14

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

30

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I PAT E N T S
Overview
The total number of AI patents published in the world
has been steadily increasing in the past two decades,
growing from 21,806 in 2000 to more than 4.5 times that,or 101,876, in 2019 (Figure 1.1.15a). The share of AI patents
published in the world exhibits a lesser increase, fromNUMBER of AI PATENT PUBLICATIONS, 2000-20

around 2% in 2000 to 2.9% in 2020 (Figure 1.1.15b). The AI
patent data is incomplete—only 8% of the dataset in 2020
includes a country or regional affiliation. There is reason
to question the data on the share of AI patent publications
by both region and geographic area, and it is therefore notincluded in the main report. See the Appendix for details.

Number of AI Patent Publications (in Thousands)

Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

100

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.15a

AI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 2000-20
2.9%

3%

2%

1%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Patent Publications (% of All Publications)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.15b

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

31

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A R X I V P U B L I CAT I O N S

Overview

In addition to the traditional avenues for
publishing academic papers (discussed
above), AI researchers have embraced thepractice of publishing their work (often
pre–peer review) on arXiv, an online
repository of electronic preprints. arXiv
allows researchers to share their findings
before submitting them to journals andconferences, which greatly accelerates
the cycle of information discovery anddissemination. The number of AI-related
publications in this section includes
preprints on arXiv under cs.AI (artificial
intelligence), cs.CL (computation andlanguage), cs.CV (computer vision), cs.NE(neural and evolutionary computing),
cs.RO (robotics), cs.LG (machine learning incomputer science), and stat.ML (machinelearning in statistics).

In just six years, the number of AI-related publications on arXiv grew
more than sixfold, from 5,478 in 2015 to 34,736 in 2020 (Figure 1.1.16).
NUMBER of AI-RELATED PUBLICATIONS on ARXIV, 2015-20
Number of AI-Related Publications on arXiv (in Thousands)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

30

20

10

0
2015

2016

2017

2018

By Region

2019
Figure 1.1.16

The analysis by region shows that while North America still holds the lead in the global share of arXiV AI-related
publications, its share has been decreasing—from 41.6% in 2017 to 36.3% in 2020 (Figure 1.1.17). Meanwhile, the share of
publications in East Asia & Pacific has grown steadily in the past five years—from 17.3% in 2015 to 26.5% in 2020.
ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by REGION, 2015-20
AI-Related Publications on arXiv (% of World Total)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

40%
36.3% North America

30%
26.5% East Asia & Paci c
22.9% Europe & Central Asia

20%

10%
4.0% South Asia
2.5% Middle East & North Africa
1.3% Latin America & Caribbean
0.3% Sub-Saharan Africa

0%
2015

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

2016

2017

2018

2019

2020

2021

Figure 1.1.17

32

2020

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
While the total number of AI-related publications on arXiv
is increasing among the three major AI powers, China iscatching up with the United States (Figure 1.1.18a andFigure 1.1.18b). The share of publication counts by theEuropean Union, on the other hand, has remained largely
unchanged.

NUMBER of AI-RELATED PUBLICATIONS on ARXIV by GEOGRAPHIC AREA, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
11,280US

Number of AI-Related Publications on arXiv

12,00010,000

8,000
6,505
EU

6,000

5,440
China

4,000

2,000

0
2015

2016

2017

2018

2019

2020
Figure 1.1.18a

ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2015-20

AI-Related Publications on arXiv (% of World Total)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

32.5% US30%

20%

18.7% EU
15.7% China

10%

0%
2015

2016

2017

2018

2019

2020
Figure 1.1.18b

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

33

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

By Field of Study
Among the six fields of study related to AI on arXiv,
the number of publications in Robotics (cs.RO) andMachine Learning in computer science (cs.LG) haveseen the fastest growth between 2015 and 2020,
increasing by 11 times and 10 times respectively(Figure 1.1.19). In 2020, cs.LG and Computer Vision(cs.CV) lead in the overall number of publications,accounting for 32.0% and 31.7%, respectively, of all AIrelated publications on arXiv. Between 2019 and 2020,
the fastest-growing categories of the seven studied
here were Computation and Language (cs.CL), by
35.4%, and cs.RO, by 35.8%.

1.1 P U B L I CAT I O N S

Among the six fields ofstudy related to AI on arXiv,
the number of publications
in Robotics (cs.RO) andMachine Learning incomputer science (cs.
LG) have seen the fastest
growth between 2015 and2020, increasing by 11 times
and 10 times respectively.

NUMBER of AI-RELATED PUBLICATIONS on ARXIV by FIELD of STUDY 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
11,001 cs.CV

Number of AI-Related Publications on arXiv

11,098 cs.LG

10,000

8,000

6,000

5,573 cs.CL

4,000
2,571 cs.RO

2,000

1,923 cs.AI
1,818 stat.ML
743 cs.NE

0
2015

2016

2017

2018

2019

2020
Figure 1.1.19

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

34

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

Deep Learning Papers on arXiv
With increased access to data and significant
improvements in computing power, the fieldof deep learning (DL) is growing at breakneck
speed. Researchers from Nesta used a topic
modeling algorithm to identify the deep learning
papers on arXiv by analyzing the abstract ofarXiv papers under the Computer Science (CS)and Machine Learning in Statistics (state.ML)
categories. Figure 1.1.20 suggests that in thelast five years alone, the overall number of DL
publications on arXiv grew almost sixfold.

NUMBER of DEEP LEARNING PUBLICATIONS on ARXIV, 2010-19
Number of Deep Learning Publications on arXiv (in Thousands)

Source: arXiv/Nesta, 2020 | Chart: 2021 AI Index Report

7

6

5

4

3

2

1
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

Figure 1.1.20

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

35

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

1. 2 C O N F E R E N C E S

Conference attendance is an indication of broader industrial and academic interest in a scientific field. In the past 20 years,
AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends inattendance at and submissions to major AI conferences.

1.2 CONFERENCESC O N F E R E N C E AT T E N DA N C E
Last year saw a significant increase in participation levels
at AI conferences, as most were offered through a virtual
format. Only the 34th Association for the Advancementof Artificial Intelligence (AAAI) Conference on Artificial
Intelligence was held in person in February 2020.
Conference organizers report that a virtual format allows
for higher attendance of researchers from all over theworld, though exact attendance numbers are difficult tomeasure.
Due to the atypical nature of 2020 conference attendance
data, the 11 major AI conferences in 2019 have been
split into two categories based on 2019 attendance data:
large AI conferences with over 3,000 attendees and small
AI conferences with fewer than 3,000 attendees. Figure
1.2.1 shows that in 2020, the total number of attendees
across nine conferences almost doubled.5 In particular,
the International Conference on Intelligent Robots
and Systems (IROS) extended the virtual conference
to allow users to watch events for up to three months,
which explains the high attendance count. Because theInternational Joint Conference on Artificial Intelligence(IJCAI) was held in 2019 and January 2021—but not in
2020—it does not appear on the charts.

Conference organizers
report that a virtual
format allows for higher
attendance of researchers
from all over the world,though exact attendance
numbers are difficult tomeasure.

5 For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference; For the KR conference,
the attendance in 2020 is based on the number of registrations; For the ICPAS conference, the attendance of 450 in 2020 is an estimate as some participants may have used anonymous Zoom accounts.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

36

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 2 C O N F E R E N C E SATTENDANCE at LARGE AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report

25,719 IROS

25,00022,011 NeurIPS

Number of Attendees

20,00015,00010,800 ICML

10,0007,500 CVPR

5,000
4,884 AAAI

2020

3,015 IJCAI

2019

2017

2016

2015

2014

2013

2012

2011

2010

0

2018

3,050 ICRA

Figure 1.2.1

ATTENDANCE at SMALL AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report
5,600
ICLR

Number of Attendees

5,000
3,972
ACL

4,000

3,726
AAMAS

3,000

2,000

1,000

469
KR
450
ICAPS

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 1.2.2

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

37

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 2 C O N F E R E N C E S

Corporate Representation at AI Research Conferences
Researchers from Virginia Tech and Ivey
Business School, Western University foundthat large technology firms have increased
participation in major AI conferences. In
their paper, titled “The De-Democratizationof AI: Deep Learning and the Compute
Divide in Artificial Intelligence Research,” theressearchers use the share of papers affiliatedwith firms over time at AI conferences toillustrate the increased presence of firmsin AI research. They argue that the unequal

distribution of compute power in academia,
which they refer to as the “compute divide,”is adding to the inequality in the era of deeplearning. Big tech firms tend to have moreresources to design AI products, but they also
tend to be less diverse than less elite or smaller
institutions. This raises concerns about bias andfairness within AI. All 10 major AI conferences
displayed in Figure 1.2.3 show an upward trend
in corporate representation, which further
extends the compute divide.

SHARE of FORTUNE GLOBAL 500 TECH-AFFILIATED PAPERS
Source: Ahmed & Wahed, 2020 | Chart: 2021 AI Index Report40%

30.8% KDD

% of Fortune Global 500 Tech-Affiliated Papers

30%

29.0% NeurIPS

28.7% ACL

25.6% ECCV

23.7% ICCV

28.5% EMNLP

20%
10%
0%
40%

27.9% ICML

30%

21.9% CVPR

20%
10%
0%
40%
30%
19.3% AAAI

20%

17.5% IJCAI

10%
0%

2000

2019

2000

2019

2000

2019

2000

2019
Figure 1.2.3

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

38

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

1. 3 A I O P E N S O U R C E S O F T WA R E
LIBRARIESA software library is a collection of computer code that is used to create applications and products. Popular AI-specific software
libraries—such as TensorFlow and PyTorch—help developers create their AI solutions quickly and efficiently. This section analyzes
the popularity of software libraries through GitHub data.

1.3 AI OPEN-SOURCESOFTWARE LIBRARIESG I T H U B S TA R S
GitHub is a code hosting platform that AI researchers anddevelopers frequently use to upload, comment on, anddownload software. GitHub users can “star” a projectto save it in their list, thereby expressing their interestsand likes—similar to the “like’’ function on Twitter andother social media platforms. As AI researchers upload
packages on GitHub that mention the use of an opensource library, the “star” function on GitHub can be used
to measure the popularity of various AI programming
open-source libraries.
Figure 1.3.1 suggests that TensorFlow (developed by
Google and publicly released in 2017) is the most popular
AI software library. The second most popular library in2020 is Keras (also developed by Google and built on topof TensorFlow 2.0). Excluding TensorFlow, Figure 1.3.2
shows that PyTorch (created by Facebook) is anotherlibrary that is becoming increasingly popular.

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

TensorFlow (developedby Google and publicly
released in 2017) is themost popular AI software
library. The second mostpopular library in 2020 is
Keras (also developed by
Google and built on topof TensorFlow 2.0).

39

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 3 A I O P E N S O U R C E S O F T WA R E
LIBRARIESNUMBER of GITHUB STARS by AI LIBRARY, 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
153 TensorFlow

Cumulative Github Stars (in Thousand)

150

100

51 Keras

50

46 PyTorch
45 Sckit-learn
31 BVLC/caffe
19 MXNet
17 Cntk
9 Theano
8 Caffe2

0
2014

2015

2016

2017

2018

2019

2020
Figure 1.3.1

NUMBER of GITHUB STARS by AI LIBRARY (excluding TENSORFLOW), 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
51 Keras

Cumulative Github Stars (in Thousand)50

46 PyTorch
45 Sckit-learn

40

31 BVLC/caffe

30

19 MXNet

20

17 Cntk

10

9 Theano
8 Caffe2

0
2014

2015

2016

2017

2018

2019

2020
Figure 1.3.2

CHAPTER 1 PREVIEWTA B L E O F C O N T E N T S

40

Artificial IntelligenceIndex Report 2021

CHAPTER 2:

Technical
Performance

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

41

C H A P T E R 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

CHAPTER 2:

Chapter Preview
Overview

43

Chapter Highlights

44

COMPUTER VISION

45

2.1 COMPUTER VISION—IMAGE

46

Image Classification

46

ImageNet

46

ImageNet: Top-1 Accuracy

46

ImageNet: Top-5 Accuracy

47

ImageNet: Training Time48

ImageNet: Training Costs

49

Highlight: Harder Tests Beyond ImageNet

50

Image Generation

51

STL-10: Fréchet Inception Distance(FID) Score

51

FID Versus Real Life

52

Deepfake Detection
Deepfake Detection Challenge (DFDC)
Human Pose Estimation
Common Objects in Context (COCO):
Keypoint Detection Challenge
Common Objects in Context (COCO):
DensePose Challenge
Semantic Segmentation

53
53
54
54

62

English Language Understanding Benchmarks

62

SuperGLUE

62

SQuAD

63

Commercial Machine Translation (MT)64

Number of Commercially Available
MT Systems

64

GPT-3

65

2.4 LANGUAGE REASONING SKILLS

67

Vision and Language Reasoning

67

Visual Question Answering (VQA) Challenge

67

Visual Commonsense Reasoning (VCR) Task

68

2.5 SPEECH

69

Speech Recognition

69

Speaker Recognition: VoxCeleb

69

Highlight: The Race Gap inSpeech Recognition Technology

71

2.2 COMPUTER VISION—VIDEO

58

Activity Recognition

58

ActivityNet

58

ActivityNet: Temporal Action
Localization Task

58

ActivityNet: Hardest Activity

59

TA B L E O F C O N T E N T S

2.3 LANGUAGE

56
57

CHAPTER 2 PREVIEW

61

55

Embodied Vision

ACCESS THE PUBLIC DATA

61

69

56

You Only Look Once (YOLO)

National Institute of Standards andTechnology (NIST) Face Recognition
Vendor Test (FRVT)Transcribe Speech: LibriSpeechCityscapes

Object Detection

Face Detection and Recognition

60
60

2.6 REASONING

72

Boolean Satisfiability Problem

72

Automated Theorem Proving (ATP)74

2.7 HEALTHCARE AND BIOLOGY76

Molecular Synthesis

76

Test Set Accuracy for Forward Chemical
Synthesis Planning

76

COVID-19 and Drug Discovery

77

AlphaFold and Protein Folding

78

EXPERT HIGHLIGHTS

79

42

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCEOV E R V I E W

Overview
This chapter highlights the technical progress in various subfields ofAI, including computer vision, language, speech, concept learning, andtheorem proving. It uses a combination of quantitative measurements,
such as common benchmarks and prize challenges, and qualitative
insights from academic papers to showcase the developments in state-ofthe-art AI technologies.
While technological advances allow AI systems to be deployed more
widely and easily than ever, concerns about the use of AI are also growing,particularly when it comes to issues such as algorithmic bias. The
emergence of new AI capabilities such as being able to synthesize images
and videos also poses ethical challenges.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

43Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCECHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• Generative everything: AI systems can now compose text, audio, and images to a sufficiently highstandard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology. That promises to generate a tremendous
range of downstream applications of AI for both socially useful and less useful purposes. It isalso causing researchers to invest in technologies for detecting generative models; the DeepFake
Detection Challenge data indicates how well computers can distinguish between different outputs.
•T he industrialization of computer vision: Computer vision has seen immense progress in the past
decade, primarily due to the use of machine learning techniques (specifically deep learning). New
data shows that computer vision is industrializing: Performance is starting to flatten on some of thelargest benchmarks, suggesting that the community needs to develop and agree on harder ones
that further test performance. Meanwhile, companies are investing increasingly large amountsof computational resources to train computer vision systems at a faster rate than ever before.
Meanwhile, technologies for use in deployed systems—like object-detection frameworks foranalysis of still frames from videos—are maturing rapidly, indicating further AI deployment.
• Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has
yielded AI systems with significantly improved language capabilities that have started to have ameaningful economic impact on the world. Google and Microsoft have both deployed the BERT
language model into their search engines, while other large language models have been developedby companies ranging from Microsoft to OpenAI. Progress in NLP has been so swift that technicaladvances have started to outpace the benchmarks to test for them. This can be seen in the rapid
emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation suite
developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
• New analyses on reasoning: Most measures of technical problems show for each time point theperformance of the best system at that time on a fixed benchmark. New analyses developed for
the AI Index offer metrics that allow for an evolving benchmark, and for the attribution to individual
systems of credit for a share of the overall performance of a group of systems over time. These
are applied to two symbolic reasoning problems, Automated Theorem Proving and Satisfiability ofBoolean formulas.
• Machine learning is changing the game in healthcare and biology: The landscape of the healthcare
and biology industries has evolved substantially with the adoption of machine learning. DeepMind’s
AlphaFold applied deep learning technique to make a significant breakthrough in the decades-long
biology challenge of protein folding. Scientists use ML models to learn representations of chemicalmolecules for more effective chemical synthesis planning. PostEra, an AI startup used ML-based
techniques to accelerate COVID-related drug discovery during the pandemic.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

44Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCECOMPUTERVISION

Computer Vision
Introduced in the 1960s, the field of computer vision has seen significant
progress and in recent years has started to reach human levels of
performance on some restricted visual tasks. Common computer
vision tasks include object recognition, pose estimation, and semantic
segmentation. The maturation of computer vision technology has unlockeda range of applications: self-driving cars, medical image analysis, consumer
applications (e.g., Google Photos), security applications (e.g., surveillance,satellite imagery analysis), industrial applications (e.g., detecting defective
parts in manufacturing and assembly), and others.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

45

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 .1 C O M P U T E R
V I S I O N — I M AG E

2.1 COMPUTER VISION—IMAGEI M AG E C L A S S I F I CAT I O N
In the 2010s, the field of image recognition andclassification began to switch from classical AI techniques
to ones based on machine learning and, specifically,
deep learning. Since then, image recognition has shiftedfrom being an expensive, domain-specific technology to
being one that is more affordable and applicable to moreareas—primarily due to advancements in the underlying
technology (algorithms, compute hardware, and theutilization of larger datasets).

ImageNet
Created by computer scientists from Stanford University
and Princeton University in 2009, ImageNet is a datasetof over 14 million images across 200 classes that expands
and improves the data available for researchers to train
AI algorithms. In 2012, researchers from the University ofToronto used techniques based on deep learning to set
a new state of the art in the ImageNet Large Scale Visual
Recognition Challenge.
Since then, deep learning techniques have ruled
the competition leaderboards—several widely usedtechniques have debuted in ImageNet competition
entries. In 2015, a team from Microsoft Research said it
had surpassed human-level performance on the image
classification task1 via the use of “residual networks”—aninnovation that subsequently proliferated into other AI
systems. Even after the end of the competition in 2017,
researchers continue to use the ImageNet dataset to test
and develop computer vision applications.
The image classification task of the ImageNet Challenge
asks machines to assign a class label to an image basedon the main object in the image. The following graphs
explore the evolution of the top-performing ImageNet
systems over time, as well as how algorithmic andinfrastructure advances have allowed researchers to

Image recognition has
shifted from being an
expensive, domain-specifictechnology to being one
that is more affordable
and applicable to moreareas—primarily due
to advancements in theunderlying technology.
increase the efficiency of training image recognition
systems and reduce the absolute time it takes to train
high-performing ones.

ImageNet: Top-1 Accuracy
Top-1 accuracy tests for how well an AI system canassign the correct label to an image, specifically whether
its single most highly probable prediction (out of all
possible labels) is the same as the target label. In recent
years, researchers have started to focus on improving
performance on ImageNet by pre-training their systems
on extra training data, for instance photos from
Instagram or other social media sources. By pre-training
on these datasets, they’re able to more effectively use
ImageNet data, which further improves performance.
Figure 2.1.1 shows that recent systems with extra training
data make 1 error out of every 10 tries on top-1 accuracy,
versus 4 errors out of every 10 tries in December 2012.
The model from the Google Brain team achieved 90.2%
on top-1 accuracy in January 2021.

1 Though it is worth noting that the human baseline for this metric comes from a single Stanford graduate student who took roughly the same test as the AI systems took.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

46

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

IMAGENET CHALLENGE: TOP-1 ACCURACY
Source: Papers with Code, 2020; AI Index, 2021 | Chart: 2021 AI Index Report

100%

90.2%, With extra training data

Top-1 Accuracy

90%

86.5%, Without extra training data

80%

70%

.

60%
80..
01/201301/201401/2015

01/2016

01/2017

01/2018

01/2019

01/2020

01/2021
Figure 2.1.1

ImageNet: Top-5 Accuracy

Top-5 accuracy asks whether the correct label is in at least the classifier’s top five predictions. Figure 2.1.2 shows thatthe error rate has improved from around 85% in 2013 to almost 99% in 2020.2
IMAGENET CHALLENGE: TOP-5 ACCURACY
Source: Papers with Code, 2020; AI Index, 2021 | Chart: 2021 AI Index Report
98.8%, With Extra Training Data

100%

97.9%, Without Extra Training Data

Top-5 Accuracy

95%

94.9% Human Performance

90%

85%

.

80%

75% Human Performance90..
01/201301/201401/2015

01/2016

01/2017

01/2018

01/2019

01/2020

01/2021
Figure 2.1.2

2 Note: For data on human error, a human was shown 500 images and then was asked to annotate 1,500 test images; their error rate was 5.1% for Top-5 classification. This is a very rough baseline, but it
gives us a sense of human performance on this task.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

47

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 .1 C O M P U T E R
V I S I O N — I M AG E

ImageNet: Training Time
Along with measuring the raw improvement in accuracy
over time, it is useful to evaluate how long it takes
to train image classifiers on ImageNet to a standard
performance level as it sheds light on advances in theunderlying computational infrastructure for large-scale
AI training. This is important to measure because thefaster you can train a system, the more quickly you
can evaluate it and update it with new data. Therefore,
the faster ImageNet systems can be trained, the moreproductive organizations can become at developing anddeploying AI systems. Imagine the difference between
waiting a few seconds for a system to train versus waitinga few hours, and what that difference means for the typeand volume of ideas researchers explore and how risky
they might be.

Imagine the difference
between waiting a few
seconds for a system totrain versus waiting a
few hours, and what thatdifference means for thetype and volume of ideas
researchers explore andhow risky they might be.

What follows are the results from MLPerf, a competition
run by the MLCommons organization that challenges
entrants to train an ImageNet network using a common(residual network) architecture, and then ranks systems
according to the absolute “wall clock” time it takes themto train a system.3

Distribution of Training Time: MLPerf does not justshow the state of the art for each competition period;
it also makes available all the data behind eachentry in each competition cycle. This, in turn, reveals
the distribution of training times for each period(Figure 2.1.3). (Note that in each MLPerf competition,competitors typically submit multiple entries that use
different permutations of hardware.)

As shown in Figure 2.1.3, the training time on ImageNet
has fallen from 6.2 minutes (December 2018) to 47
seconds (July 2020). At the same time, the amount of
hardware used to achieve these results has increased
dramatically; frontier systems have been dominated by
the use of “accelerator” chips, starting with GPUs in the2018 results, and transitioning to Google’s TPUs for the
best-in-class results from 2019 and 2020.

Figure 2.1.4 shows that in the past couple of years,
training times have shortened, as has the variance
between MLPerf entries. At the same time, competitors
have started to use larger and larger numbers ofaccelerator chips to speed training times. This is in linewith broader trends in AI development, as large-scaletraining becomes better understood, with a higher
degree of shared best practices and infrastructure.

3 The next MLPerf update is planned for June 2021.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

48

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

IMAGENET: TRAINING TIME and HARDWARE of the BESTSYSTEM

IMAGENET: DISTRIBUTION of TRAINING TIME
Source: MLPerf, 2020 | Chart: 2021 AI Index Report

Source: MLPerf, 2020 | Chart: 2021 AI Index Report

10,000

6.2 Min

4,000

6
1,000

4
4,096

2,000

3

Training Time

3,000

Training Time (mins)

5
Number of Accelerators

2 .1 C O M P U T E R
V I S I O N — I M AG E

100

2

1.3 Min

10

1,000
1,024

47 Sec

1

1

640

0

0
12/2018

06/2019

Number of Accelerators

07/2020

2018

2019

2020

Training Time
Figure 2.1.3

Figure 2.1.4

ImageNet: Training Costs
How much does it cost to train a contemporary imagerecognition system? The answer, according to tests run by
the Stanford DAWNBench team, is a few dollars in 2020,
down by around 150 times from costs in 2017 (Figure

2.1.5). To put this in perspective, what cost one entrant
around USD 1,100 to do in October 2017 now costs about
USD 7.43. This represents progress in algorithm design aswell as a drop in the costs of cloud-computing resources.

IMAGENET: TRAINING COST (to 93% ACCURACY)
Source: DAWNBench, 2020 | Chart: 2021 AI Index Report

2,000

Cost (U.S. Dollars; Log Scale)

1,000
500
200
100
50
20
10

$7.43

5
2
1
12/201706/2018

12/2018

06/2019

12/2019
Figure 2.1.5

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

49

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

Harder Tests Beyond ImageNet
In spite of the progress in performance on ImageNet, current computer vision systems are still not
perfect. To better study their limitations, researchers have in recent years started to develop morechallenging image classification benchmarks. But since ImageNet is already a large dataset, which
requires a nontrivial amount of resources to use, it does not intuitively make sense to simply expand theresolution of the images in ImageNet or the absolute size of the dataset—as either action would further
increase the cost to researchers when training systems on ImageNet. Instead, people have tried to figureout new ways to test the robustness of image classifiers by creating custom datasets, many of which arecompatible with ImageNet (and are typically smaller). These include
IMAGENET ADVERSARIAL:
This is a dataset of images
similar to those found in
ImageNet but incorporating
natural confounders (e.g., abutterfly sitting on a carpet witha similar texture to the butterfly),and images that are persistently
misclassified by contemporary
systems. These images “cause
consistent classification
mistakes due to scene
complications encounteredin the long tail of scene
configurations and by exploitingclassifier blind spots,” according
to the researchers. Therefore,making progress on ImageNet
Adversarial could improve theability of models to generalize.

IMAGENET-C:
This is a dataset of common
ImageNet images with 75 visual
corruptions applied to them(e.g., changes in brightness,
contrast, pixelations, fog
effects, etc.). By testing systems
against this, researchers canprovide even more information
about the generalization
capabilities of these models.

IMAGENET-RENDITION:
This tests generalization byseeing how well ImageNettrained models can categorize
30,000 illustrations of 200
ImageNet classes. Since
ImageNet is designed to be built
out of photos, generalization
here indicates that systems
have learned something more
subtle about what they’re trying
to classify, because they’re able
to “understand” the relationshipbetween illustrations and thephotographed images they’vebeen trained on.

What is the Time Table for Tracking This Data? As these benchmarks are relatively new, the plan is to
wait a couple of years for the community to test a range of systems against them, which will generate
the temporal information necessary to make graphs tracking progress overtime.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

50CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

I M AG E G E N E R AT I O NImage generation is the task of generating images
that look indistinguishable from “real” images. Image
generation systems have a variety of uses, ranging fromaugmenting search capabilities (it is easier to search fora specific image if you can generate other images likeit) to serving as an aid for other generative uses (e.g.,
editing images, creating content for specific purposes,
generating multiple variations of a single image to help
designers brainstorm, and so on).
In recent years, image generation progress has
accelerated as a consequence of the continued
improvement in deep learning–based algorithms, as wellas the use of increased computation and larger datasets.

STL-10: Fréchet Inception Distance (FID) Score
One way to measure progress in image generation is via atechnique called Fréchet Inception Distance (FID), whichroughly correlates to the difference between how a given
AI system “thinks” about a synthetic image versus a real
image, where a real image has a score of 0 and synthetic
images that look similar have scores that approach 0.
Figure 2.1.6 shows the progress of generative models
over the past two years at generating convincing
synthetic images in the STL-10 dataset, which is designed
to test how effective systems are at generating images
and gleaning other information about them.

STL-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE
Source: Papers with Code, 2020 | Chart: 2021 AI Index Report

Fréchet Inception Distance (FID) Score

45

40

35

30

25.4

25

20
01/2018

04/2018

07/2018

10/2018

01/2019

04/2019

07/2019

10/2019

01/2020

04/2020

07/2020
Figure 2.1.6

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

51

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

FID Versus Real Life
FID has drawbacks as an evaluation technique—specifically, it assesses progress on image generation
via quantitative metrics that use data from the model
itself, rather than other evaluation techniques. Anotherapproach is using teams of humans to evaluate theoutputs of these models; for instance, the Human eYe
Perceptual Evaluation (HYPE) method tries to judge image
quality by showing synthetically generated images tohumans and using their qualitative ratings to drive theevaluation methodology. This approach is more expensive
and slower to run than typical evaluations, but it may
become more important as generative models get better.

Qualitative Examples: To get a sense of progress, you
can look at the evolution in the quality of syntheticallygenerated images over time. In Figure 2.1.7, you can
see the best-in-class examples of synthetic images ofhuman faces, ordered over time. By 2018, performance of
this task had become sufficiently good that it is difficultfor humans to easily model further progress (though it
is possible to train machine learning systems to spot
fakes, it is becoming more challenging). This provides avisceral example of recent progress in this domain andunderscores the need for new evaluation methods togauge future progress. In addition, in recent years people
have turned to doing generative modeling on a broader
range of categories than just images of people’s faces,
which is another way to test for generalization.

GAN PROGRESS ON FACE GENERATION
Source: Goodfellow et al., 2014; Radford et al., 2016; Liu & Tuzel, 2016; Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al., 2020; AI Index, 2021

2014

2015

2016

2017
2018
2020
Figure 2.1.7

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

52

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

D E E P FA K E D E T E CT I O NAdvances in image synthesis have created newopportunities as well as threats. For instance, in recent
years, researchers have harnessed breakthroughsin synthetic imagery to create AI systems that cangenerate synthetic images of human faces, thensuperimpose those faces onto the faces of other people
in photographs or movies. People call this applicationof generative technology a “deepfake.” Malicious usesof deepfakes include misinformation and the creationof (predominantly misogynistic) pornography. To try
to combat this, researchers are developing deepfakedetection technologies.

Deepfake Detection Challenge (DFDC)
Created in September 2019 by Facebook, the Deepfake
Detection Challenge (DFDC) measures progress on
deepfake-detection technology. A two-part challenge,
DFDC asks participants to train and test their models froma public dataset of around 100,000 clips. The submissions
are scored on log loss, a classification metric based on
probabilities. A smaller log loss means a more accurate
prediction of deepfake videos. According to Figure
2.1.8, log loss dropped by around 0.5 as the challenge
progressed between December 2019 and March 2020.

DEEPFAKE DETECTION CHALLENGE: LOG LOSS
Source: Kaggle, 2020 | Chart: 2021 AI Index Report

0.7

0.6

Log Loss

0.5

0.4

0.3

0.2

0.19

0.1
0.0
1/6/2020

1/16/2020

1/26/2020

2/5/2020

2/15/2020

2/25/2020

3/6/2020

3/16/2020

3/26/2020
Figure 2.1.8

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

53

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

H U M A N P O S E E S T I M AT I O N
Human pose estimation is the problem of estimating thepositions of human body parts or joints (wrists, elbows,etc.) from a single image. Human pose estimation is aclassic “omni-use” AI capability. Systems that are good atthis task can be used for a range of applications, such as
creating augmented reality applications for the fashionindustry, analyzing behaviors gleaned from physicalbody analysis in crowds, surveilling people for specificbehaviors, aiding with analysis of live sporting andathletic events, mapping the movements of a person to a
virtual avatar, and so on.

2 .1 C O M P U T E R
V I S I O N — I M AG E

Common Objects in Context (COCO):
Keypoint Detection Challenge
Common Objects in Context (COCO) is a large-scale
dataset for object detection, segmentation, andcaptioning with 330,000 images and 1.5 million object
instances. Its Keypoint Detection Challenge requires
machines to simultaneously detect an object or a person
and localize their body keypoints—points in the image
that stand out, such as a person’s elbows, knees, andother joints. The task evaluates algorithms based on
average precision (AP), a metric that can be used tomeasure the accuracy of object detectors. Figure 2.1.9
shows that the accuracy of algorithms in this task has
improved by roughly 33% in the past four years, with thelatest machine scoring 80.8% on average precision.

COCO KEYPOINT CHALLENGE: AVERAGE PRECISION
Source: COCO Leaderboard, 2020 | Chart: 2021 AI Index Report

90%

Average Precision (AP)

80%

80.8%

70%

60%

50%
07/2016

01/2017

07/2017

01/2018

07/2018

01/2019

07/2019

01/2020

07/2020
Figure 2.1.9

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

54

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .1 C O M P U T E R
V I S I O N — I M AG E

Common Objects in Context (COCO):
DensePose Challenge
DensePose, or dense human pose estimation, is the taskof extracting a 3D mesh model of a human body from a
2D image. After open-sourcing a system called DensePosein 2018, Facebook built DensePose-COCO, a large-scale
dataset of image-to-surface correspondences annotatedon 50,000 COCO images. Since then, DensePose has
become a canonical benchmark dataset.
The COCO DensePose Challenge involves tasks ofsimultaneously detecting people, segmenting theirbodies, and estimating the correspondences between
image pixels that belong to a human body and atemplate 3D model. The average precision is calculated
based on the geodesic point similarity (GPS) metric,a correspondence matching score that measures thegeodesic distances between the estimated points andthe true location of the body points on the image. The
accuracy has grown from 56% in 2018 to 72% in 2019
(Figure 2.1.10).

COCO DENSEPOSE CHALLENGE: AVERAGE PRECISION
Source: arXiv & CodaLab, 2020 | Chart: 2021 AI Index Report

75%
72%

Average Precision (AP)

70%

65%

60%

55%

50%
03/2018

05/2018

07/2018

09/2018

11/2018

01/2019

03/2019

05/2019

07/2019

09/2019
Figure 2.1.10

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

55

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

S E M A N T I C S E G M E N TAT I O N
Semantic segmentation is the task of classifying eachpixel in an image to a particular label, such as person,
cat, etc. Where image classification tries to assign alabel to the entire image, semantic segmentation tries toisolate the distinct entities and objects in a given image,allowing for more fine-grained identification. Semantic
segmentation is a basic input technology for self-drivingcars (identifying and isolating objects on roads), image
analysis, medical applications, and more.

Cityscapes
Cityscapes is a large-scale dataset of diverse urban street
scenes across 50 different cities recorded during thedaytime over several months (during spring, summer, andfall) of the year. The dataset contains 5,000 images withhigh-quality, pixel-level annotations and 20,000 weekly
labeled ones. Semantic scene understanding, especially

2 .1 C O M P U T E R
V I S I O N — I M AG E

in the urban space, is crucial to the environmental
perception of autonomous vehicles. Cityscapes is usefulfor training deep neural networks to understand the urban
environment.
One Cityscapes task that focuses on semantic
segmentation is the pixel-level semantic labeling task.
This task requires an algorithm to predict the per-pixel
semantic labeling of the image, partitioning an image
into different categories, like cars, buses, people, trees,
and roads. Participants are evaluated based on theintersection-over-union (IoU) metric. A higher IoU scoremeans a better segmentation accuracy. Between 2014 and2020, the mean IoU increased by 35% (Figure 2.1.11). There
was a significant boost to progress in 2016 and 2017 whenpeople started using residual networks in these systems.

CITYSCAPES CHALLENGE: PIXEL-LEVEL SEMANTIC LABELING TASK
Source: Papers with Code, 2020 | Chart: 2021 AI Index Report90%

Mean Intersection-Over-Union (mIoU)

85.1% With Extra Training Data

85%
82.3% Without Extra Training Data

80%

75%

70%

65%

60%
01/2015

01/2016

01/2017

01/2018

01/2019

01/2020
Figure 2.1.11

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

56

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 .1 C O M P U T E R
V I S I O N — I M AG E

EMBODIED VISION
The performance data so far shows that computer visionsystems have advanced tremendously in recent years.
Object recognition, semantic segmentation, and humanpose estimation, among others, have now achieved
significant levels of performance. Note that these visualtasks are somewhat passive or disembodied. Thatis, they can operate on images or videos taken fromcamera systems that are not physically able to interact
with the surrounding environment. As a consequenceof the continuous improvement in those passive tasks,
researchers have now started to develop more advanced
AI systems that can be interactive or embodied—that
is, systems that can physically interact with and modify
the surrounding environment in which they operate: forexample, a robot that can visually survey a new building
and autonomously navigate it, or a robot that can learn
to assemble pieces by watching visual demonstrations
instead of being manually programmed for this.
Progress in this area is currently driven by thedevelopment of sophisticated simulation environments,
where researchers can deploy robots in virtual spaces,
simulate what their cameras would see and capture, anddevelop AI algorithms for navigation, object search, andobject grasping, among other interactive tasks. Becauseof the relatively early nature of this field, there are few
standardized metrics to measure progress. Instead, here
are brief highlights of some of the available simulators,
their year of release, and any other significant feature.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S• Thor (AI2, 2017) focuses on sequential abstract
reasoning with predefined “magic” actions that are
applicable to objects.
• Gibson (Stanford, 2018) focuses on visual navigationin photorealistic environments obtained with 3D
scanners.
• iGibson (Stanford, 2019) focuses on full interactivity
in large realistic scenes mapped from real houses andmade actable: navigation + manipulation (known in
robotics as “mobile manipulation”).
• AI Habitat (Facebook, 2019) focuses on visualnavigation with an emphasis on much faster
execution, enabling more computationally expensive
approaches.
• ThreeDWorld (MIT and Stanford, 2020) focuses onphotorealistic environments through game engines,
as well as adds simulation of flexible materials, fluids,
and sounds.
• SEAN-EP (Yale, 2020) is a human-robot interaction
environment with simulated virtual humans thatenables the collection of remote demonstrations fromhumans via a web browser.
• Robosuite (Stanford and UT Austin, 2020) is a modular
simulation framework and benchmark for robot
learning.

57

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2.2 COMPUTERV I S I O N —V I D E O

Video analysis is the task of making inferences over sequential image frames, sometimes with the inclusion of an audio feed.
Though many AI tasks rely on single-image inferences, a growing body of applications require computer vision machines to reason
about videos. For instance, identifying a specific dance move benefits from seeing a variety of frames connected in a temporalsequence; the same is true of making inferences about an individual seen moving through a crowd, or a machine carrying out asequence of movements over time.

2.2 COMPUTER VISION—VIDEOAC T I V I T Y R E C O G N I T I O N
The task of activity recognition is to identify various
activities from video clips. It has many important
everyday applications, including surveillance by video
cameras and autonomous navigation of robots. Researchon video understanding is still focused on short events,
such as videos that are a few seconds long. Longer-term
video understanding is slowly gaining traction.

ActivityNet
Introduced in 2015, ActivityNet is a large-scale video
benchmark for human-activity understanding. The
benchmark tests how well algorithms can label andcategorize human behaviors in videos. By improving
performance on tasks like ActivityNet, AI researchers aredeveloping systems that can categorize more complex

behaviors than those that can be contained in a single
image, like characterizing the behavior of pedestrians on
a self-driving car’s video feed or providing better labelingof specific movements in sporting events.

ActivityNet: Temporal Action Localization Task
The temporal action localization task in the ActivityNet
challenge asks machines to detect time segments in a
600-hour, untrimmed video sequence that contains several
activities. Evaluation on this task focuses on (1) localization:
how well can the system localize the interval with the precise
start time and end time; and (2) recognition: how well can
the system recognize the activity and classify it into thecorrect category (such as throwing, climbing, walking thedog, etc.). Figure 2.2.1 shows that the highest mean average
precision of the temporal action localization task among
submissions has grown by 140% in the last five years.

ACTIVITYNET: TEMPORAL ACTION LOCALIZATION TASK
Source: ActivityNet, 2020 | Chart: 2021 AI Index Report

50%

42.8%

Mean Average Precision (mAP)

40%

30%

20%

10%

0%
2016

2017

2018

2019

2020
Figure 2.2.1

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

58

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2.2 COMPUTERV I S I O N —V I D E O

ActivityNet: Hardest Activity
Figure 2.2.2 shows the hardest activities of the temporal
action location task in 2020 and how their mean average
precision compares with the 2019 result. Drinking coffee
remained the hardest activity in 2020. Rock-paper-

scissors, though still the 10th hardest activity, saw thegreatest improvement among all activities, increasing by
129.2%—from 6.6% in 2019 to 15.22% in 2020.

ACTIVITYNET: HARDEST ACTIVITIES, 2019-20
Source: ActivityNet, 2020 | Chart: 2021 AI Index Report2019

Drinking coffee

2020

High jump
Polishing furniture
Putting in contactlensesRemoving curlers
Rock-paper-scissors
Running a marathon
Shot put
Smoking a cigarette
Throwing darts
0%

2%

4%

6%

8%

10%

Mean Average Precision (mAP)CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

12%

14%
Figure 2.2.2

59

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2.2 COMPUTERV I S I O N —V I D E O

O B J E C T D E T E CT I O N
Object detection is the task of identifying a given object
in an image. Frequently, image classification and image
detection are coupled together in deployed systems.
One way to get a proxy measure for the improvement
in deployed object recognition systems is to study theadvancement of widely used object detection systems.

You Only Look Once (YOLO)You Only Look Once (YOLO) is a widely used open source
system for object detection, so its progress has been
included on a standard task on YOLO variants to give asense of how research percolates into widely used open
source tools. YOLO has gone through multiple iterations

since it was first published in 2015. Over time, YOLO has
been optimized along two constraints: performance andinference latency, as shown in Figure 2.2.3. What thismeans, specifically, is that by measuring YOLO, one canmeasure the advancement of systems that might nothave the best absolute performance but are designed
around real-world needs, like low-latency inference
over video streams. Therefore, YOLO systems might
not always contain the absolute best performance as
defined in the research literature, but they will represent
good performance when faced with trade-offs such asinference time.

YOU ONLY LOOK ONCE (YOLO): MEAN AVERAGE PRECISION
Source: Redmon & Farhadi (2016 & 2018), Bochkovskiy et al. (2020), Long et al. (2020) | Chart: 2021 AI Index Report80

Mean Average Precision (mAP50)

70
65.2

60

PP-YOLO(Resolution: 608)

04/2020

11/2020

YOLOv3(Resolution: 608)

50

40

YOLOv4(Resolution: 608)

YOLOv2(Resolution: Unclear)30

20
12/2016

04/2018

Figure 2.2.3

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

60

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2.2 COMPUTERV I S I O N —V I D E O

FAC E D E T E CT I O N A N D
RECOGNITION
Facial detection and recognition is one of the use-cases
for AI that has a sizable commercial market and has
generated significant interest from governments andmilitaries. Therefore, progress in this category gives
us a sense of the rate of advancement in economicallysignificant parts of AI development.

National Institute of Standards and Technology(NIST) Face Recognition Vendor Test (FRVT)The Face Recognition Vendor Tests (FRVT) by the
National Institute of Standards and Technology (NIST)
provide independent evaluations of commerciallyavailable and prototype face recognition technologies.
FRVT measures the performance of automated facerecognition technologies used for a wide range of civiland governmental tasks (primarily in law enforcement
and homeland security), including verification of visa
photos, mug shot images, and child abuse images.
Figure 2.2.4 shows the results of the top-performing 1:1
algorithms measured on false non-match rate (FNMR)
across several different datasets. FNMR refers to therate at which the algorithm fails when attempting tomatch the image with the individual. Facial recognition
technologies on mug-shot-type and visa photos have
improved the most significantly in the past four years,
falling from error rates of close to 50% to a fraction of apercent in 2020.4

NIST FRVT 1:1 VERIFICATION ACCURACY by DATASET, 2017-20
Source: National Institute of Standards and Technology, 2020 | Chart: 2021 AI Index Report

1

False Non-Match rate (FNMR; Log Scale)

0.5

0.2
0.1
0.05
0.0293, WILD Photos FNMR @ FMR ≤ 0.00001

0.02
0.01
0.0064, BORDER Photos FNMR @ FMR = 0.000001

0.005

0.0035, VISABORDER Photos FNMR@FMR ≤ 0.000001
0.0025, VISA Photos FNMR @ FMR ≤ 0.000001
0.0023, MUGSHOT Photos FNMR @ FMR ≤ 0.00001 DT>=12 YRS
0.0022, MUGSHOT Photos FNMR @ FMR ≤ 0.00001

0.002
2017

2018

2019

2020

2021

2022
Figure 2.2.4

4 You can view details and examples of various datasets on periodically updated FRVT 1:1 verification reports.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

61

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 . 3 L A N G UAG E

Natural language processing (NLP) involves teaching machines to interpret, classify, manipulate, and generate language.
From the early use of handwritten rules and statistical techniques to the recent adoption of generative models and deeplearning, NLP has become an integral part of our lives, with applications in text generation, machine translation, question
answering, and other tasks.

2.3 LANGUAGE
In recent years, advances in natural language processing
technology have led to significant changes in large-scale
systems that billions of people access. For instance, inlate 2019, Google started to deploy its BERT algorithm
into its search engine, leading to what the company said
was a significant improvement in its in-house quality
metrics. Microsoft followed suit, announcing later in 2019that it was using BERT to augment its Bing search engine.

E N G L I S H L A N G UAG E
U N D E R S TA N D I N G B E N C H M A R KS
SuperGLUE
Launched in May 2019, SuperGLUE is a single-metricbenchmark that evaluates the performance of a model ona series of language understanding tasks on establisheddatasets. SuperGLUE replaced the prior GLUE benchmark(introduced in 2018) with more challenging and diverse tasks.
The SuperGLUE score is calculated by averaging scores on
a set of tasks. Microsoft’s DeBERTa model now tops theSuperGLUE leaderboard, with a score of 90.3, comparedwith an average score of 89.8 for SuperGLUE’s “humanbaselines.” This does not mean that AI systems havesurpassed human performance on all SuperGLUE tasks, but
it does mean that the average performance across the entire
suite has exceeded that of a human baseline. The rapid paceof progress (Figure 2.3.1) suggests that SuperGLUE mayneed to be made more challenging or replaced by harder
tests in the future, just as SuperGLUE replaced GLUE.

SUPERGLUE BENCHMARK
Source: SuperGLUE Leaderboard, 2020 | Chart: 2021 AI Index Report

100

90

90.3 Score

89.8 Human Performance

Score

80

70

60

50

07/2019

07/2019

08/2019

10/2019

01/2020

12/2020

01/2021

01/2021
Figure 2.3.1

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

62

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 . 3 L A N G UAG E

SQuAD
The Stanford Question Answering Dataset, or SQuAD,
is a reading-comprehension benchmark that measures
how accurately a NLP model can provide short answers
to a series of questions pertaining to a small articleof text. The SQuAD test makers established a humanperformance benchmark by having a group of people
read Wikipedia articles on a variety of topics and thenanswer multiple-choice questions about those articles.
Models are given the same task and are evaluated on
the F1 score, or the average overlap between the model
prediction and the correct answer. Higher scores indicate
better performance.
Two years after the introduction of the original SQuAD,
in 2016, SQuAD 2.0 was developed once the initialbenchmark revealed increasingly fast performances by

the participants (mirroring the trend seen in GLUE andSuperGLUE). SQuAD 2.0 combines the 100,000 questionsin SQuAD 1.1 with over 50,000 unanswerable questions
written by crowdworkers to resemble answerable ones.
The objective is to test how well systems can answer
questions and to determine when systems know that noanswer exists.
As Figure 2.3.2 shows, the F1 score for SQuAD 1.1
improved from 67.75 in August 2016 to surpass humanperformance of 91.22 in September 2018—a 25-month
period—whereas SQuAD 2.0 took just 10 months to beathuman performance (from 66.3 in May 2018 to 89.47
in March 2019). In 2020, the most advanced models of
SQuAD 1.1 and SQuAD 2.0 reached the F1 scores of 95.38
and 93.01, respectively.

SQUAD 1.1 and SQUAD 2.0: F1 SCORE
Source: CodaLab Worksheets, 2020 | Chart: 2021 AI Index Report100
95.4 SQuAD 1.1
93.0 SQuAD 2.0
91.2 Human 1.1

F1 Score

90 89.5 Human 2.0

80

70

60
07/2016

01/2017

07/2017

01/2018

07/2018

01/2019

07/2019

01/2020
Figure 2.3.2

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

63

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCEC O M M E R C I A L M AC H I N E
T R A N S L AT I O N ( M T )Number of Commercially Available MT
Systems

Machine translation (MT), the subfield of computational
linguistics that investigates the use of software totranslate text or speech from one language to another,has seen significant improvement due to advances inmachine learning. Recent progress in MT has prompted
developers to shift from symbolic approaches toward
ones that use both statistical and deep learning
approaches.

2 . 3 L A N G UAG E

The trend in the number of commercially available
systems speaks to the significant growth of commercial
machine translation technology and its rapid adoption
in the commercial marketplace. In 2020, the number ofcommercially available independent cloud MT systems
with pre-trained models increased to 28, from 8 in2017, according to Intento, a startup that evaluatescommercially available MT services (Figure 2.3.3).

NUMBER of INDEPENDENT MACHINE TRANSLATION SERVICES

Number of Independent Machine Translation Services

Source: Intento, 2020 | Chart: 2021 AI Index Report

Preview
25

Commercial

20

15

10

5

0
0

05/2017

07/2017

11/2017

03/2018

07/2018

12/2018

06/2019

11/2019

07/2020
Figure 2.3.3

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

64

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

G P T-3
In July 2020, OpenAI unveiled GPT-3, the largest known
dense language model. GPT-3 has 175 billion parameters
and was trained on 570 gigabytes of text. For comparison,
its predecessor, GPT-2, was over 100 times smaller, at
1.5 billion parameters. This increase in scale leads tosurprising behavior: GPT-3 is able to perform tasks it
was not explicitly trained on with zero to few training
examples (referred to as zero-shot and few-shot learning,respectively). This behavior was mostly absent in themuch smaller GPT-2. Furthermore, for some tasks (butnot all; e.g., SuperGLUE and SQuAD2), GPT-3 outperforms
state-of-the-art models that were explicitly trained tosolve those tasks with far more training examples.

2 . 3 L A N G UAG E

Figure 2.3.4, adapted from the GPT-3 paper,
demonstrates the impact of scale (in terms of model
parameters) on task accuracy (higher is better) in zero-,one-, and few-shot learning regimes. Each point on the
curve corresponds to an average performance accuracy,
aggregated across 42 accuracy-oriented benchmarks. As
model size increases, average accuracy in all task regimes
increases accordingly. Few-shot learning accuracy
increases more rapidly with scale, compared with zeroshot learning, which suggests that large models canperform surprisingly well given minimal context.

GPT-3: AVERAGE PERFORMANCE across 42 BENCHMARKS

Accuracy

Source: OpenAI (Brown et al.), 2020 | Chart: 2021 AI Index Report

60%

57.4%
Few-Shot

50%

51.0%
One-Shot

40%

42.6%
Zero-Shot

30%

20%

10%
0%
1

10

100

Number of Parameters (Billions)Figure 2.3.4

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

65

Artificial IntelligenceIndex Report 2021

That a single model can achieve state-of-the-art or close
to state-of-the-art performance in limited-training-data
regimes is impressive. Most models until now have been
designed for a single task, and thus can be evaluatedeffectively by a single metric. In light of GPT-3, we
anticipate novel benchmarks that are explicitly designed
to evaluate zero- to few-shot learning performancefor language models. This will not be straightforward.
Developers are increasingly finding model novel
capabilities (e.g., the ability to generate a website from atext description) that will be difficult to define, let alone
measure performance on. Nevertheless, the AI Index iscommitted to tracking performance in this new context
as it evolves.
Despite its impressive capabilities, GPT-3 has several
shortcomings, many of which are outlined in the original
paper. For example, it can generate racist, sexist, andotherwise biased text. Furthermore, GPT-3 (and otherlanguage models) can generate unpredictable andfactually inaccurate text. Techniques for controllingand “steering” such outputs to better align with humanvalues are nascent but promising. GPT-3 is also expensive
to train, which means that only a limited number oforganizations with abundant resources can currently
afford to develop and deploy such models. Finally, GPT-3
has an unusually large number of uses, from chatbots
to computer code generation to search. Future users arelikely to discover more applications, both good and bad,making it difficult to identify the range of possible usesand forecast their impact on society.

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 . 3 L A N G UAG E

That a single model canachieve state-of-theart or close to state-ofthe-art performance inlimited-training-data
regimes is impressive.
Most models until now
have been designed for asingle task, and thus canbe evaluated effectively
by a single metric.

Nevertheless, research to address harmful outputs anduses is ongoing at several universities and industrial
research labs, including OpenAI. For more details, refer
to work by Bender and Gebru et al. and the proceedings
from a recent Stanford Institute for Human-Centered
Artificial Intelligence (HAI) workshop (which included
researchers from OpenAI), “Understanding theCapabilities, Limitations, and Societal Impact of Large
Language Models.”

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

66

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .4 L A N G UAG EREASONING
SKILLS

2.4 LANGUAGE REASONING SKILLSV I S I O N A N D L A N G UAG EREASONING
Vision and language reasoning is a research area that
addresses how well machines jointly reason about visualand text data.

Visual Question Answering (VQA) Challenge
The VQA challenge, introduced in 2015, requires
machines to provide an accurate natural language
answer, given an image and a natural language question
about the image based on a public dataset. Figure 2.4.1

shows that the accuracy has grown by almost 40% since
its first installment at the International Conference on
Computer Vision (ICCV) in 2015. The highest accuracy ofthe 2020 challenge is 76.4%. This achievement is closer tothe human baseline of 80.8% accuracy and represents a
1.1% absolute increase in performance from the top 2019
algorithm.

VISUAL QUESTION ANSWERING (VQA) CHALLENGE: ACCURACY
Source: VQA Challenge, 2020 | Chart: 2021 AI Index Report
80.8% Human Baseline

80%
76.4% Accuracy

Accuracy

75%

70%

65%

60%

55%
10/2015

04/2016

10/2016

04/2017

10/2017

04/2018

10/2018

04/2019

10/2019

04/2020
Figure 2.4.1

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

67

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .4 L A N G UAG EREASONING
SKILLS

Visual Commonsense Reasoning (VCR) Task
The Visual Commonsense Reasoning (VCR) task,
first introduced in 2018, asks machines to answer achallenging question about a given image and justify
that answer with reasoning (whereas VQA just requests
an answer). The VCR dataset contains 290,000 pairs ofmultiple-choice questions, answers, and rationales, aswell as over 110,000 images from movie scenes.
The main evaluation mode for the VCR task is the Q->ARscore, requiring machines to first choose the right answer(A) to a question (Q) among four answer choices (Q->A)and then select the correct rationale (R) among four
rationale choices based on the answer. A higher score isbetter, and human performance on this task is measured
by a QA->R score of 85. The best-performing machine has
improved on the Q->AR score from 44 in 2018 to 70.5 in2020 (Figure 2.4.2), which represents a 60.2% increase inperformance from the top competitor in 2019.

VISUAL COMMONSENSE REASONING (VCR) TASK: Q->AR Score
Source: VCR Leaderboard, 2020 | Chart: 2021 AI Index Report

100

85 Human Performance80

Q->AR Score

70.5

60

40

20
11/2018 01/2019

03/2019

05/2019

07/2019

09/2019

11/2019

01/2020

03/2020

05/2020

07/2020
Figure 2.4.2

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

68Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2.5 SPEECH

A major aspect of AI research is the analysis and synthesis of human speech conveyed via audio data. In recent years,
machine learning approaches have drastically improved performance across a range of tasks.

2.5 SPEECHSPEECH RECOGNITION
Speech recognition, or automatic speech recognition (ASR),
is the process that enables machines to recognize spoken
words and convert them to text. Since IBM introduced its
first speech recognition technology in 1962, the technology
has evolved with voice-driven applications such as
Amazon Alexa, Google Home, and Apple Siri becoming
increasingly prevalent. The flexibility and predictive powerof deep neural networks, in particular, has allowed speech
recognition to become more accessible.

Transcribe Speech: LibriSpeech
LibriSpeech is a dataset, first introduced in 2015, made up
of 1,000 hours of speech from audiobooks. It has become
widely used for the development and testing of speech
recognition technologies. In recent years, neural-networkbased AI systems have started to dramatically improve
performance on LibriSpeech, lowering the word error rate
(WER; 0% is optimal performance) to around 2% (Figure
2.5.1a and Figure 2.5.1b).
Developers can test out their systems on LibriSpeech in
two ways:• Test Clean determines how well their systems cantranscribe speech from a higher-quality subset of the
LibriSpeech dataset. This test gives clues about how
well AI systems might perform in more controlledenvironments.
• Test Other determines how systems can deal withlower-quality parts of the LibriSpeech dataset. This
test suggests how well AI systems might perform innoisier (and perhaps more realistic) environments.
There has been substantial progress recently on bothdatasets, with an important trend emerging in the past two
years: The gap between performance on Test Clean and Test
Other has started to close significantly for frontier systems,

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

shifting from an absolute performance difference of morethan seven points in late 2015 to a difference of less than
one point in 2020. This reveals dramatic improvements in
the robustness of ASR systems over time and suggests that
we might be saturating performance on LibriSpeech—in
other words, harder tests may be needed.

Speaker Recognition: VoxCeleb
Speaker identification tests how well machine learning
systems can attribute speech to a particular person. The
VoxCeleb dataset, first introduced in 2017, contains overa million utterances from 6,000 distinct speakers, and its
associated speaker-identification task tests the error rate
for systems that try to attribute a particular utterance toa particular speaker. A better (lower) score in VoxCeleb
provides a proxy for how well a machine can distinguish
one voice among 6,000. Evaluation method for VoxCeleb is
Equal Error Rate (EER), a commonly used metric for identity
verification systems. EER provides a measure for both thefalse positive rate (assigning a label incorrectly) and thefalse negative rate (failing to assign a correct label).
In recent years, progress on this task has come fromhybrid systems—systems that fuse contemporary deeplearning approaches with more structured algorithms,
developed by the broader speech-processing community.
As of 2020, error rates have dropped such that computers
have a very high (99.4%) ability to attribute utterances toa given speaker (Figure 2.5.2)
Still, obstacles remain: These systems face challenges
processing speakers with different accents and indifferentiating among speakers when confronted with alarge dataset (it is harder to identify one person in a setof a billion people than to pick out one person across the
VoxCeleb training set of 6,000).

69

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2.5 SPEECHLIBRISPEECH: WORD ERROR RATE, TEST CLEAN

LIBRISPEECH: WORD ERROR RATE, TEST OTHER

Source: Papers with Code, 2020 | Chart: 2021 AI Index Report

Source: Papers with Code, 2020 | Chart: 2021 AI Index Report

6

14
12

Word Error Rate (WER)

Word Error Rate (WER)

5

4

3

2

10

8

6

4
1.4

1

2.6

2

0

0
01/2016

01/2017

01/2018

01/2019

01/2020

01/2016

01/2017

Figure 2.5.1a01/2018

01/2019

01/2020
Figure 2.5.1b

VOXCELEB: EQUAL ERROR RATE
Source: VoxCeleb, 2020 | Chart: 2021 AI Index Report8%

Equal Error Rate (EER)

6%

4%

2%

0.6%

0%
2017

2018

2019

2020
Figure 2.5.2

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

70

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report

Average Word Error Rate (WER)

40%

30%

20%

10%

0%

The results suggest that, on average,
systems made 19 errors every
hundred words for white speakers
and 35 errors for Black speakers—
nearly twice as many. Moreover,the systems performed particularly
poorly for Black men, with more than
40 errors for every hundred words(Figure 2.5.3). The breakdown by
ASR systems shows that gaps are
similar across companies (Figure
2.5.4). This research emphasizes
the importance of addressing thebias of AI technologies and ensuring
equity as they become mature andBlack Men Black Women White Men White Women
deployed.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

Source: Koenecke et al., 2020 | Char

Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report

40%

Average Word Error Rate (WER)

Average Word Error Rate (WER)

25%

30%

20%

10%

20%

15%

10%

5%
0%

0%
Black MenBlack WomenApple

White Men White Women
Av..

100%
Figure 2.5.3

TESTINGS on LEADING SPEECH-to-TEXT SERVICES:
WORD ERROR RATE by SERVICE and RACE, 2019
Source: Koenecke et al., 2020 | Chart: 2021 AI Index Report

Black Speakers

25%
Average Word Error Rate (WER)

Researchers from Stanford University
found that state-of-the-art ASR
systems exhibited significant
racial and gender disparity—they
misunderstand Black speakers
twice as often as white speakers.
In the paper, titled “Racial
Disparities in Automated Speech
Recognition,” authors ran thousandsof audio snippets of white andBlack speakers, transcribed frominterviews conducted with 42 white
speakers and 73 Black speakers,
through leading speech-to-text
services by Amazon, Apple, Google,
TESTINGS on LEADING SPEECH-to-TEXT SERVICES:
IBM,
andWORD ERROR
RATE
byMicrosoft.
RACE and GENDER, 2019TESTINGS on LEADIN
WORD ERROR RATE bTESTINGS on LEADING SPEECH-to-TEXT SERVICES:
WORD ERROR RATE by RACE and GENDER, 2019White Speakers

20%

15%

10%

5%
0%
Apple
Av..

The Race Gap inSpeech Recognition
Technology

2.5 SPEECH

100%

IBM

GoogleAmazonMicrosoft
Figure 2.5.4

71Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 .6 R E A S O N I N GThis section measures progress on symbolic (or logical) reasoning in AI, which is the process of drawing conclusions fromsets of assumptions. We consider two major reasoning problems, Boolean Satisfiability (SAT) and Automated Theorem
Proving (ATP). Each has real-world applications (e.g., circuit design, scheduling, software verification, etc.) and posessignificant measurement challenges. The SAT analysis shows how to assign credit for the overall improvement in the field toindividual systems over time. The ATP analysis shows how to measure performance given an evolving test set.
All analyses below are original to this report. Lars Kotthoff wrote the text and performed the analysis for the SAT section.
Geoff Sutcliffe, Christian Suttner, and Raymond Perrault wrote the text and performed the analysis for the ATP section. This
work had not been published at the time of writing; consequently, a more academically rigorous version of this section (withreferences, more precise details, and further context) is included in the Appendix.

2.6 REASONINGBOOLEAN SATISFIABILIT Y PROBLEM
Analysis and text by Lars Kotthoff
The SAT problem considers whether there is an
assignment of values to a set of Boolean variables, joined
by logical connectives, that makes the logical formula it
represents true. Many real-world problems, such as circuitdesign, automated theorem proving, and scheduling, canbe represented and solved efficiently as SAT problems.
The performance of the top-, median-, and bottom-ranked
SAT solvers was examined from each of the last five years(2016–2020) of the SAT Competition, which has beenrunning for almost 20 years, to measure a snapshot ofstate-of-the-art performance. In particular, all 15 solvers
were run on all 400 SAT instances from the main track of
the 2020 competition and the time (in CPU seconds) it took
to solve all instances was measured.5 Critically, each solver
was run on the same hardware, such that comparisonsacross years would not be confounded by improvements
in hardware efficiency over time.
While performance of the best solvers from 2016 to2018 did not change significantly, large improvements
are evident in 2019 and 2020 (Figure 2.6.1). These
improvements affect not only the best solvers but also
their competitors. The performance of the median-ranked
solver in 2019 is better than that of the top-ranked solvers

in all previous years, and the performance of the medianranked solver in 2020 is almost on par with the top-ranked
solver in 2019.
Performance improvements in SAT—and more generally,
hard computational AI problems—come primarily
from two areas of algorithmic improvements: novel
techniques and more efficient implementations of existing
techniques. Typically, performance improvements arise
primarily from novel techniques. However, more efficient
implementations (which can arise with performance
improvements in hardware over time) can also increase
performance. Therefore, it is difficult to assess whether
performance improvements arise primarily from novel
techniques or more efficient implementations. To address
this problem, the temporal Shapley value, which is thecontribution of an individual system to state-of-the-art
performance over time, was measured (see the Appendix
for more details).
Figure 2.6.2 shows the temporal Shapley value
contributions of each solver for the different competition
years. Note that the contributions of the solvers in 2016
are highest because there is no previous state-of-the-art
to compare them with in our evaluation and that theircontribution is not discounted.

5 Acknowledgments: The Advanced Research Computing Center at the University of Wyoming provided resources for gathering the computational data. Austin Stephen performed the computational
experiments.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

72

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .6 R E A S O N I N GTOTAL TIME to SOLVE ALL 400 INSTANCES for EACH SOLVER and YEAR (LOWER IS BETTER), 2016-20
Source: Kotthoﬀ, 2020 | Chart: 2021 AI Index Report

2.00e+06
1.75e+06

YalSAT 03r

YalSAT

CCAnrSim

PauSatTotal Time (CPUs)

1.50e+06

glu_vc1.20e+06

MapleLCMDistChronoBT

glue_alt
Candy

1.00e+06 MapleCOMSPS_DRUP

Maple_LCM_Dist

smallsat
expMC_VSIDS_LRB_Switch_2500
MapleLCMDiscChronoBT-DL-v3

Maple_mix

8.00e+05
7.00e+05

Kissat-sc2020-sat

2016

2017

2018

2019

2020
Figure 2.6.1

TEMPORAL SHAPLEY VALUE CONTRIBUTIONS of INDIVIDUAL SOLVERS to the STATE of the ART OVER TIME (HIGHERIS BETTER), 2016-20
Source: Kotthoﬀ, 2020 | Chart: 2021 AI Index Report

1.00e+06

MapleCOMSPS_DRUP
glue_alt

Temporal Shapley Value

1.00e+05

YalSAT 03r

Candy
Maple_LCM_Dist
glu_vc1.00e+04

smallsat
expMC_VSIDS_LRB_Switch_2500

MapleLCMDistChronoBT

Kissat-sc2020-sat

CCAnrSim

MapleLCMDiscChronoBT-DL-v3
PauSatMaple_mix

1.00e+03

1.00e+02

YalSAT

2016

2017

2018

2019

2020
Figure 2.6.2

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

73

Artificial IntelligenceIndex Report 2021

According to the temporal Shapley value, in 2020 the best
solver contributes significantly more than the medianand bottom-ranked solvers do. The 2020 winner, Kissat,
has the highest temporal Shapley value of any solvers
excluding the first year. The changes it incorporates,
compared with those of previous solvers, are almost
exclusively more efficient data structures and algorithms;
Kissat thus impressively demonstrates the impact ofgood engineering on the state-of-the-art performance.
By contrast, smallsat, the solver with the largest
temporal Shapley value (but not the winner) in 2019,focuses on improved heuristics instead of a more efficient
implementation. The same is true of Candy, the solver
with the largest temporal Shapley value in 2017, whose
main novelty is to analyze the structure of a SAT instanceand apply heuristics based on this analysis. Interestingly,
neither solver ranked first in their respective years; bothwere outperformed by versions of the Maple solver,which nevertheless contributes less to the state of theart. This indicates that incremental improvements,
while not necessarily exciting, are important for good
performance in practice.
Based on our limited analysis of the field, novel
techniques and more efficient implementations havemade equally important contributions to the stateof the art in SAT solving. Incremental improvementsof established solvers are as likely to result in top
performance as more substantial improvements ofsolvers without a long track record.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

CHAPTER 2:
T E C H N I CA LPERFORMANCE

2 .6 R E A S O N I N G

AUTOMATED THEOREM PROVING(ATP)
Analysis and text by Christian Suttner, Geoff Sutcliffe, andRaymond Perrault
Automated Theorem Proving (ATP) concerns thedevelopment and use of systems that automate soundreasoning, or the derivation of conclusions that follow
inevitably from facts. ATP systems are at the heartof many computational tasks, including software
verification. The TPTP problem library was used toevaluate the performance of ATP algorithms from 1997 to2020 and to measure the fraction of problems solved byany system over time (see the Appendix for more details).
The analysis extends to the whole TPTP (over 23,000
problems) in addition to four salient subsets (eachranging between 500 and 5,500 problems)—clausenormal form (CNF), first-order form (FOF), monomorphic
typed first-order form (TF0) with arithmetic, andmonomorphic typed higher-order form (TH0) theorems—
all including the use of the equality operator.
Figure 2.6.3 shows that the fraction of problems solved
climbs consistently, indicating progress in the field. The
noticeable progress from 2008 to 2013 included strong
progress in the FOF, TF0, and TH0 subsets. In FOF, which
has been used in many domains (e.g., mathematics,
real-world knowledge, software verification), there weresignificant improvements in the Vampire, E, and iProversystems. In TF0 (primarily used for solving problems inmathematics and computer science) and TH0 (useful insubtle and complex topics such as philosophy and logic),
there was rapid initial progress as systems developed
techniques that solved “low-hanging fruit” problems. In
2014–2015, there was another burst of progress in TF0,
as the Vampire system became capable of processing
TF0 problems. It is noteworthy that, since 2015, progresshas continued but slowed, with no indication of rapid
advances or breakthroughs in the last few years.

74CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .6 R E A S O N I N GPERCENTAGE of PROBLEMS SOLVED, 1997-2020
Source: Suttner, Sutcliﬀe & Perrault, 2020 | Chart: 2021 AI Index Report

80%
71.4% Tf0 Thm Equ Ari
64.0% Th0 Thm Equ Nar
61.1% Fof Thm Rfo Seq% of Problems Solved

60%

57.8% Cnf Uns Rfo Seq Nhn
49.2% All

40%

20%

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

0%

Figure 2.6.3

While this analysis demonstrates progress in ATP, there
is obviously room for much more. Two keys to solve
ATP problems are axiom selection (given a large set of
axioms, only some of which are needed for a proof ofthe conjecture, how to select an adequate subset ofthe axioms); and search choice (at each stage of an ATP
system’s search for a solution, which logical formula(e)should be selected for attention). The latter issue has
been at the forefront of ATP research since its inception

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

in the 1960s, while the former has become increasingly
important as large bodies of knowledge are encoded
for ATP. In the last decade, there has been growing useof machine learning approaches to addressing thesetwo key challenges (e.g., in the MaLARea and Enigma
ATP systems). Recent results from the CADE ATP System
Competition (CASC) have shown that the emergence ofmachine learning is a potential game-changer for ATP.

75

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .7 H E A LT H CA R E
A N D B I O LO GY

2.7 HEALTHCARE AND BIOLOGY
In collaboration with the “State of AI Report”

MOLECULAR SYNTHESISText by Nathan Benaich and Philippe Schwaller
Over the last 25 years, the pharmaceutical industry has
shifted from developing drugs from natural sources (e.g.,plants) to conducting large-scale screens with chemicallysynthesized molecules. Machine learning allows scientists
to determine what potential drugs are worth evaluating in
the lab and the most effective way of synthesizing them.
Various ML models can learn representations of chemicalmolecules for the purposes of chemical synthesis planning.
A way to approach chemical synthesis planning is torepresent chemical reactions with a text notation andcast the task as a machine translation problem. Recent
work since 2018 makes use of the transformer architecture
trained on large datasets of single-step reactions. Laterwork in 2020 approached model forward prediction andretrosynthesis as a sequence of graph edits, where thepredicted molecules were built from scratch.

Notably, these approaches offer an avenue to rapidly
sweep through a list of candidate drug-like molecules insilico and output synthesizability scores and synthesis
plans. This enables medicinal chemists to prioritize
candidates for empirical validation and could ultimately
let the pharmaceutical industry mine the vast chemical
space to unearth novel drugs to benefit patients.

Test Set Accuracy for Forward Chemical
Synthesis Planning
Figure 2.7.1 shows the top-1 accuracy of models
benchmarked on a freely available dataset of one million
reactions in the U.S. patents.6 Top-1 accuracy means thatthe product predicted by the model with the highest
likelihood corresponds to the one that was reported
in the ground truth. Data suggests that progress inchemical synthesis planning has seen steady growth in
the last three years, as the accuracy grew by 15.6% in2020 from 2017. The latest molecular transformer scored
92% on top-1 accuracy in November 2020.

CHEMICAL SYNTHESIS PLANS BENCHMARK: TOP-1 TEST ACCURACY
Source: Schwaller, 2020 | Chart: 2021 AI Index Report

100%

Transformer | SMILES
GNN-based | Graph Edits

Seq2seq with attn |SMILESTransformer | SMILES

GNN-based | Graph Edits
76.5%
76.5%Best
BestHuman
Human

Top-1 Accuracy

75%

50%

25%

0%
12/201706/2018

11/2018

Human Prediction Benchmark

08/2019
Standard 500k Benchmark

11/2020
Figure 2.7.1

Human Prediction
BenchmarkStandard
500k atBenchmark6 Acknowledgment:
Philippe Schwaller
IBM Research–Europe and the University of Bern provided instructions and resources for gathering and analyzing the data.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

76

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .7 H E A LT H CA R E
A N D B I O LO GY

COVID -19 AND DRUG DISCOVERY
AI-powered drug discovery has gone open source tocombat the COVID-19 pandemic. COVID Moonshot is a
crowdsourced initiative joined by over 500 international
scientists to accelerate the development of a COVID-19
antiviral. The consortium of scientists submits theirmolecular designs pro bono, with no claims. PostEra,an AI startup, uses machine learning and computational
tools to assess how easily compounds can be madeusing the submissions from the scientists and generates
synthetic routes. After the first week, Moonshot received
over 2,000 submissions, and PostEra designed syntheticroutes in under 48 hours. Human chemists would havetaken three to four weeks to accomplish the same task.

Figure 2.7.2 shows the accumulated number ofsubmissions by scientists over time. Moonshot received
over 10,000 submissions from 365 contributors aroundthe world in just four months. Toward the end of August
2020, the crowdsourcing had served its purpose, and theemphasis moved to optimize the lead compounds and set
up for animal testing. As of February 2021, Moonshot aims
to nominate a clinical candidate by the end of March.

POSTERA: TOTAL NUMBER of MOONSHOT SUBMISSIONS
Source: PostEra, 2020 | Chart: 2021 AI Index Report
15,545

Total Number of Moonshot Submissions

15,00010,0005,000

003/2020

04/2020

05/2020

06/2020

07/2020

08/2020

09/2020

10/2020

11/2020

12/2020
Figure 2.7.2

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

77

CHAPTER 2:
T E C H N I CA LPERFORMANCE

Artificial IntelligenceIndex Report 2021

2 .7 H E A LT H CA R E
A N D B I O LO GY

ALPHAFOLD AND PROTEIN FOLDING
The protein folding problem, a grand challenge instructural biology, considers how to determine the threedimensional structure of proteins (essential componentsof life) from their one-dimensional representations(sequences of amino acids7). A solution to this problem
can have wide ranging applications—from better
understanding the cellular basis of life, to fueling drug
discovery, to curing diseases, to engineering de-novo
proteins for industrial tasks, and more.
In recent years, machine learning-based approaches
have started to make a meaningful difference on theprotein folding problem. Most notably, DeepMind’s
AlphaFold debuted in 2018 at the Critical Assessmentof Protein Structure (CASP) competition, a biennialcompetition to foster and measure progress on protein
folding. At CASP, competing teams are given amino acidsequences and tasked to predict the three-dimensional
structures of the corresponding proteins, the latter ofwhich are determined through laborious and expensive
experimental methods (e.g., nuclear magnetic resonancespectroscopy, X-ray crystallography, cryo-electron
microscopy, etc.) and unknown to the competitors.
Performance on CASP is commonly measured by the
Global Distance Test (GDT) score, a number between 0
and 100, which measures the similarity between two
protein structures. A higher GDT score is better.
Figure 2.7.3, adapted from the DeepMind blog post,shows the median GDT scores of the best team on someof the harder types of proteins to predict (the ‘freemodelling’ category of proteins) at CASP over the last
14 years. In the past, winning algorithms were typically
based on physics based models; however, in the last twocompetitions, Deepmind’s AlphaFold and AlphaFold 2algorithms achieved winning scores through the partial
incorporation of deep learning techniques.

CASP: MEDIAN ACCURACY of PREDICTIONS in FREE-MODELING by THE BEST TEAM, 2006-20
Source: DeepMind, 2020 | Chart: 2021 AI Index Report

100

Global Distance Test (GDT_TS)

ALPHAFOLD 2
80

ALPHAFOLD

60

40

20

0
CASP7 (2006)

CASP8 (2008)

CASP9 (2010)

CASP10 (2012)

CASP11 (2014)

CASP12 (2016)

CASP13 (2018) CASP14 (2020)
Figure 2.7.3

7 Currently most protein folding algorithms leverage multiple sequence alignments—many copies of a protein sequence representing the same protein across evolution—rather than just asingle sequence.

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

78

Artificial IntelligenceIndex Report 2021

CHAPTER 2:
T E C H N I CA LPERFORMANCE

EXPERTHIGHLIGHTS

EXPERT HIGHLIGHTSThis year, the AI Index asked AI experts to share their thoughts on the most significant technical AI breakthroughs
in 2020. Here’s a summary of their responses, along with a couple of individual highlights.

What was the single most impressive AI advancement in 2020?
•T he two most mentioned systems by a significant margin were AlphaFold (DeepMind), a model for molecular
assay, and GPT-3 (OpenAI), a generative text model.

What single trend will define AI in 2021?
•E xperts predict that more advances will be built by using pretrained models. For instance, GPT-3 is alarge NLP model that can subsequently be fine-tuned for excellent performance on specific, narrow tasks.
Similarly, 2020 saw various computer vision advancements built on top of models pretrained on very large
image datasets.

What aspect of AI technical progress, deployment, and development are you most excited tosee in 2021?
• “ It’s interesting to note the dominance of the Transformers architecture, which started for machinetranslation but has become the de facto neural network architecture. More broadly, whereas NLP trailed
vision in terms of adoption of deep learning, now it seems like advances in NLP are also driving vision.” —Percy Liang, Stanford University• “ The incredible recent advancements in language generation have had a profound effect on the fields of NLP
and machine learning, rendering formerly difficult research challenges and datasets suddenly useless while
simultaneously encouraging new research efforts into the fascinating emergent capabilities (and importantfailings) of these complex new models.” —Carissa Schoenick, Allen Institute of AI Research

CHAPTER 2 PREVIEWTA B L E O F C O N T E N T S

79

Artificial IntelligenceIndex Report 2021

CHAPTER 3:

The Economy

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

80

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

CHAPTER 3:

Chapter Preview
Overview

82

3.3 CORPORATE ACTIVITY

98

Chapter Highlights

83

Industry Adoption

98

Global Adoption of AI

98

3.1 JOBS84

AI Adoption by Industry and Function

99

AI Hiring

84

Type of AI Capabilities Adopted

99

AI Labor Demand

86

Consideration and Mitigation of
Risks from Adopting AI

101

The Effect of COVID-19

103

Industrial Robot Installations

104

Global AI Labor Demand

86

U.S. AI Labor Demand: By Skill Cluster 87
U.S. Labor Demand: By Industry

88

U.S. Labor Demand: By State90

AI Skill Penetration

91

Global Comparison

91

Global Comparison: By Industry

92

3.2 INVESTMENT

93

Corporate Investment

93

Startup Activity

94

Global Trend

94

Regional Comparison

95

Focus Area Analysis

97

Global Trend

104

Regional Comparison

105

Earnings Calls

106

ACCESS THE PUBLIC DATACHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

81

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMYOV E R V I E W

Overview
The rise of artificial intelligence (AI) inevitably raises the question of how muchthe technologies will impact businesses, labor, and the economy more generally.
Considering the recent progress and numerous breakthroughs in AI, the field offers
substantial benefits and opportunities for businesses, from increasing productivity
gains with automation to tailoring products to consumers using algorithms,
analyzing data at scale, and more.
However, the boost in efficiency and productivity promised by AI also presents
great challenges: Companies must scramble to find and retain skilled talent tomeet their production needs while being mindful about implementing measures tomitigate the risks of using AI. Moreover, the COVID-19 pandemic has caused chaosand continued uncertainty for the global economy. How have private companies
relied on and scaled AI technologies to help their business navigate through thismost difficult time?
This chapter looks at the increasingly intertwined relationship between AI and theglobal economy from the perspective of jobs, investment, and corporate activity.
It first analyzes the worldwide demand for AI talent using data on hiring rates
and skill penetration rates from LinkedIn as well as AI job postings from Burning
Glass Technologies. It then looks at trends in private AI investment using statistics
from S&P Capital IQ (CapIQ), Crunchbase, and Quid. The third, final section
analyzes trends in the adoption of AI capabilities across companies, trends in robot
installations across countries, and mentions of AI in corporate earnings, drawing
from McKinsey’s Global Survey on AI, the International Federation of Robotics(IFR), and Prattle, respectively.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

82

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• “Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI
investment in 2020, with more than USD 13.8 billion, 4.5 times higher than 2019.
• Brazil, India, Canada, Singapore, and South Africa are the countries with the highest
growth in AI hiring from 2016 to 2020. Despite the COVID-19 pandemic, the AI hiringcontinued to grow across sample countries in 2020.
• More private investment in AI is being funneled into fewer startups. Despite the pandemic,
2020 saw a 9.3% increase in the amount of private AI investment from 2019—a higher
percentage increase than from 2018 to 2019 (5.7%), though the number of newly funded
companies decreased for the third year in a row.
• Despite growing calls to address ethical concerns associated with using AI, efforts toaddress these concerns in the industry are limited, according to a McKinsey survey. For
example, issues such as equity and fairness in AI continue to receive comparatively little
attention from companies. Moreover, fewer companies in 2020 view personal or individual
privacy risks as relevant, compared with in 2019, and there was no change in the percentageof respondents whose companies are taking steps to mitigate these particular risks.
• Despite the economic downturn caused by the pandemic, half the respondents in a
McKinsey survey said that the coronavirus had no effect on their investment in AI, while
27% actually reported increasing their investment. Less than a fourth of businesses
decreased their investment in AI.
• The United States recorded a decrease in its share of AI job postings from 2019 to 2020—
the first drop in six years. The total number of AI jobs posted in the United States also
decreased by 8.2%, from 325,724 in 2019 to 300,999 in 2020.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

83

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3.1 J O B S

Attracting and retaining skilled AI talent is challenging. This section examines the latest trend in AI hiring, labor demand,and skill penetration, with data from LinkedIn and Burning Glass.

3.1 JOBSAI HIRING
How rapidly is the growth of AI jobs in different countries?
This section first looks at LinkedIn data that gives theAI hiring rate for different countries. The AI hiring rateis calculated as the number of LinkedIn members who
include AI skills on their profile or work in AI-related
occupations and who added a new employer in the samemonth their new job began, divided by the total number ofLinkedIn members in the country. This rate is then indexed
to the average month in 2016; for example, an index of 1.05
in December 2020 points to a hiring rate that is 5% higher
than the average month in 2016. LinkedIn makes monthto-month comparisons to account for any potential lags
in members updating their profiles. The index for a year isthe average index over all months within that year.

This data suggests that the hiring rate has been
increasing across all sample countries in 2020. Brazil,
India, Canada, Singapore, and South Africa are thecountries with the highest growth in AI hiring from 2016
to 2020 (Figure 3.1.1). Across the 14 countries analyzed,
the AI hiring rate in 2020 was 2.2 times higher, on
average, than that in 2016. For the top country, Brazil,
the hiring index grew by more than 3.5 times. Moreover,despite the COVID-19 pandemic, AI hiring continued its
growth across the 14 sampled countries in 2020 (Figure
3.1.2).
For more explorations of cross-country comparisons, see
the AI Index Global AI Vibrancy Tool.

AI HIRING INDEX by COUNTRY, 2020
Source: LinkedIn, 2020 | Chart: 2021 AI Index Report
Brazil
India
CanadaSingapore
South Africa
Germany
Australia
United States
Argentina
United Kingdom
TurkeyItaly
France
China

0

1

2
AI Hiring Index

3
Figure 3.1.1

1 Countries included are a sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were also included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as in other countries, and should be interpreted accordingly.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

84

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3.1 J O B S

AI HIRING INDEX by COUNTRY, 2016-20
Source: LinkedIn, 2020 | Chart: 2021 AI Index Report

3.4 Brazil

3

2.8 India

2.7 Canada

2.3 South Africa

2.2 Germany

2.1 Australia

2.1 United States

2.0 Argentina

1.8 United Kingdom

1.8 Turkey

1.7 Italy

2.5 Singapore

2
1
0
3

AI Hiring Index

2
1
0
3
2
1
0
3
1.6 France

2

1.3 China

1
0

2016

2020

2016

2020

2016

2020

2016

2020
Figure 3.1.2

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

85

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMYAI LABOR DEMAND
This section analyzes the AI labor demand based ondata from Burning Glass, an analytics firm that collects
postings from over 45,000 online job sites. To develop
a comprehensive, real-time portrait of labor market
demand, Burning Glass aggregated job postings,
removed duplicates, and extracted data from job posting
text. Note that Burning Glass updated the data coveragein 2020 with more job sites; as a result, the numbers inthis report should not be directly compared with data inthe 2019 report.

Global AI Labor Demand
Demand for AI labor in six countries covered by Burning
Glass data—the United States, the United Kingdom,
Canada, Australia, New Zealand, and Singapore—has

3.1 J O B S

grown significantly in the last seven years (Figure 3.1.3).
On average, the share of AI job postings among all job
postings in 2020 is more than five times larger than in
2013. Of the six countries, Singapore exhibits the largest
growth, as its percentage of AI job postings across all job
roles in 2020 is 13.5 times larger than in 2013.
The United States is the only country among the six thatrecorded a decrease in its share of AI job postings from2019 to 2020—the first drop in six years. This may be dueto the coronavirus pandemic or the country’s relatively
more mature AI labor market. The total number of AI jobs
posted in the United States also decreased by 8.2%, from
325,724 in 2019 to 300,999 in 2020.

AI JOB POSTINGS (% of ALL JOB POSTINGS) by COUNTRY, 2013-20
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report

AI Job Postings (% of All Job Postings)

2.5%

2.4% Singapore

2.0%

1.5%

1.0%
0.8% United States
0.8% United Kingdom
0.7% Canada

0.5%

0.5% Australia
0.2% New Zealand

0.0%
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 3.1.3

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

86Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

U.S. AI Labor Demand: By Skill Cluster
Taking a closer look at the AI labor demand in the UnitedStates between 2013 and 2020, Figure 3.1.4 breaks down
demand during that period year by year according to skill
cluster. Each skill cluster consists of a list of AI-related
skills; for example, the neural network skill cluster includes
skills like deep learning and convolutional neural network.
The Economy chapter appendix provides a complete list of
AI skills under each skill cluster.
Between 2013 and 2020, AI jobs related to machinelearning and artificial intelligence experienced the fastest
growth in online AI job postings in the United States,
increasing from 0.1% of total jobs to 0.5% and 0.03% to
0.3%, respectively. As noted earlier, 2020 shows a decreasein the share of AI jobs among overall job postings across all
skill clusters.

3.1 J O B S

Between 2013 and2020, AI jobs related tomachine learning andartificial intelligenceexperienced the fastest
growth in online AI job
postings in the UnitedStates, increasing from
0.1% of total jobs to
0.5% and 0.03% to 0.3%,
respectively.

AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by SKILL CLUSTER, 2013-20
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report

0.5% Machine Learning

AI Job Postings (% of All Job Postings)

0.5%

0.4%

0.3%

0.3% Artificial Intelligence

0.2%
0.1% Neural Networks
0.1% Natural Language Processing

0.1%

0.1% Robotics
0.1% Visual Image Recognition
0.1% Autonomous Driving

0.0%
2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 3.1.4

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

87Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3.1 J O B S

In 2020, industries
focused on information(2.8%); professional,
scientific, and technicalservices (2.5%); andagriculture, forestry,
fishing, and hunting(2.1%) had the highest
share of AI job postings
among all job postings in
the United States.

U.S. Labor Demand: By Industry
To dive deeper into how AI job demand in the U.S. labormarket varies across industries, this section looks at theshare of AI job postings across all jobs posted in the UnitedStates by industry in 2020 (Figure 3.1.5) as well as thetrend in the past 10 years (Figure 3.1.6).
In 2020, industries focused on information (2.8%);
professional, scientific, and technical services (2.5%); andagriculture, forestry, fishing, and hunting (2.1%) had thehighest share of AI job postings among all job postings
in the United States. While the first two have always
dominated demand for AI jobs, the agriculture, forestry,
fishing, and hunting industry saw the biggest jump—byalmost 1 percentage point—in the share of AI jobs from2019 to 2020.

AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by INDUSTRY, 2020
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report

Information
Professional, Scientific, Tech Services
Agriculture, Forestry, Fishing, Hunting
Manufacturing
Finance and Insurance
Public Administration
Mining, Quarrying, Oil/Gas Extraction
Management of Companies/Enterprises
Educational Services
Wholesale Trade
Utilities
Real Estate and Rental and Leasing
Other Services
Retail Trade
Transportation and Warehousing
Health Care and Social AssistanceConstruction
Arts, Entertainment, and Recreation
Accommodation and Food Services
0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

AI Job Postings (% of All Job Postings)
Figure 3.1.5

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

88

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3.1 J O B S

AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by INDUSTRY, 2013-20
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report
2.8%
Information

3.0%

2.5%
Professional, Scientific, Tech Services

2.1%
Agriculture, Forestry, Fishing, Hunting

1.8%
Manufacturing

2.0%
1.0%
0.0%
3.0%
1.4%
Finance and Insurance

AI Job Postings (% of All Job Postings)

2.0%

0.9%
Public Administration

1.0%

0.8%
Mining, Quarrying, Oil/Gas Extraction

0.8%
Management of Companies/Enterprises

0.0%
3.0%
2.0%

0.8%
Educational Services

1.0%

0.7%
Wholesale Trade

0.6%
Utilities
0.5%
Other Services

0.0%
3.0%
2.0%
1.0%

0.5%
Real Estate and Rental and Leasing

0.4%
Retail Trade

0.2%
Transportation and Warehousing

0.2%
Health Care and Social Assistance

0.2%
Construction

0.2%
Arts, Entertainment, and Recreation

0.0%
3.0%
2.0%
1.0%
0.0%

2013

2020

2013

2020

0.1%
Accommodation and Food Services

2013

2020

2013

2020
Figure 3.1.6

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

89Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3.1 J O B S

U.S. Labor Demand: By StateAs the competition for AI talent intensifies, where arecompanies seeking employees with machine learning, data
science, and other AI-related skills within the United States?
Figure 3.1.7 examines the labor demand by U.S. state in2020, plotting the share of AI job postings across all job
postings on the y-axis and the total number of AI jobs
posted on a log scale on the x-axis. The chart shows thatthe District of Columbia has the highest share of AI jobs
posted (1.88%), overtaking Washington state in 2019; andCalifornia remains the state with the highest number of AI
job postings (63,433).
In addition to Washington, D.C., six states registeredover 1% of AI job postings among all job postings—
Washington, Virginia, Massachusetts, California, New York,and Maryland—compared with five last year. California
also has more AI job postings than the next three states
combined, which are Texas (22,539), New York (18,580),
and Virginia (17,718).

AI JOB POSTINGS (TOTAL and % of ALL JOB POSTINGS) by U.S. STATE and DISTRICT, 2020
Source: Burning Glass, 2020 | Chart: 2021 AI Index Report

2.0%
District of Columbia

AI Job Postings (% of All Job Postings)

Washington
Virginia

1.5%

Massachusetts
California
New York

1.0%

MarylandNew Jersey
Delaware
New Mexico

North Carolina

Arizona

Connecticut

Illinois
Pennsylvania

South Dakota
HawaiiVermont

0.5%

Maine

Wyoming

Wisconsin
Indiana

Arkansas

Texas

Michigan Ohio

Alabama

Idaho

Montana
AlaskaOregonMissouri

FloridaLouisiana

0.0%
100

200

500

1,000

2,000

5,000

10,00020,00050,000 100,000

Number of AI Job Postings (Log Scale)Figure 3.1.7

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

90Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

A I S K I L L P E N E T R AT I O N
How prevalent are AI skills across occupations? The AI skill
penetration metric shows the average share of AI skills
among the top 50 skills in each occupation, using LinkedIn
data that includes skills listed on a member’s profile,positions held, and the locations of the positions.

Global Comparison
For cross-country comparison, the relative penetration
rate of AI skills is measured as the sum of the penetrationof each AI skill across occupations in a given country,
divided by the average global penetration of AI skills
across the same occupations. For example, a relative

3.1 J O B S

penetration rate of 2 means that the average penetrationof AI skills in that country is 2 times the global average
across the same set of occupations.
Among the sample countries shown in Figure 3.1.8, theaggregated data from 2015 to 2020 shows that India (2.83
times the global average) has the highest relative AI skill
penetration rate, followed by the United States (1.99 times
the global average), China (1.40 times the global average),
Germany (1.27 times the global average), and Canada (1.13
times the global average).2

RELATIVE AI SKILL PENETRATION RATE by COUNTRY, 2015-20
Source: LinkedIn, 2020 | Chart: 2021 AI Index Report

India
United States
China
Germany
CanadaSouth Korea
Singapore
United Kingdom
FranceAustralia
Brazil
Italy
South Africa
0

1

2

3

Relative AI Skill Penetration Rate
Figure 3.1.8

2 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

91Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

Global Comparison: By Industry

3.1 J O B S

industries, while the United States and China frequently
appear high up on the list. Other pockets of specialization
worth highlighting with relative skill penetration rates ofmore than 1 include Germany in hardware and networking
as well as manufacturing; and Israel in manufacturing andeducation.

To provide an in-depth sectoral decomposition of AI skill
penetration across industries and sample countries,
Figure 3.1.9 includes the aggregated data of the top fiveindustries with the highest AI skill penetration globallyin the last five years: education, finance, hardware andnetworking, manufacturing, and software and IT.3 India
has the highest relative AI skill penetration across all five

RELATIVE AI SKILL PENETRATION RATE by INDUSTRY, 2015-20
Source: LinkedIn, 2020 | Chart: 2021 AI Index Report

Education

Finance

Hardware & Networking

India

India

India

United States

United States

United States

South Korea

CanadaGermany

IsraelSwitzerland

Germany
United Kingdom

China
IsraelGermanySouth Africa

FranceChina

Netherlands

Singapore

0

1

2

3

Manufacturing

0

India

India
United States

Germany

China

IsraelSouth Korea

United Kingdom

Germany

Spain

CanadaChina

Israel
1

2

3

2

3

0

1

2

3

Software & IT Services

United States

0

1

2

3

0

1

Relative AI Skill Penetration Rate
Figure 3.1.9

3 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

92

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 2 I N V E S T M E N T

This section explores the investment activity of private companies by NetBase Quid based on data from CapIQ and
Crunchbase. Specifically, it looks at the latest trends in corporate AI investment, such as private investment, public
offerings, mergers and acquisitions (M&A), and minority stakes related to AI. The section then focuses on the private
investment in AI, or how much private funding goes into AI startups and which sectors are attracting significant
investment and in which countries.

3.2 INVESTMENTC O R P O R AT E I N V E S T M E N T
The total global investment in AI, including private
investment, public offerings, M&A, and minority stakes,
increased by 40% in 2020 relative to 2019 for a total of
USD 67.9 billion (Figure 3.2.1). Given the pandemic, manysmall businesses have suffered disproportionately. As aresult, industry consolidation and increased M&A activity

in 2020 are driving up the total corporate investment in
AI. M&A made up the majority of the total investment
amount in 2020, increasing by 121.7% relative to 2019.
Several high-profile acquisitions related to AI took
place in 2020, including NVIDIA’s acquisition of Mellanox
Technologies and Capgemini’s of Altran Technologies.

GLOBAL CORPORATE INVESTMENT in AI by INVESTMENT ACTIVITY, 2015-20
Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

Total Investment (in Millions of U.S. Dollars)

70,00067,854

Private Investment
60,000Public O ering
Merger/Acquisition

50,000Minority Stake

48,85144,07542,23843,81140,00023,00230,00038,65936,57620,0004,140

17,69912,751

10,000

7,952

0

4,328

2015

13,0972016

18,932

19,849

2017

4,493

8,541

2018

2019

2020
Figure 3.2.1

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

93Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 2 I N V E S T M E N T

S TA R T U P ACT I V I T Y

40,00030,00020,00010,000

2020

2019

2018

2017

0
2016

More private investment in AI is being funneledinto fewer startups. Despite the pandemic,
2020 saw a 9.3% increase in the amount ofprivate AI investment from 2019—a higher
percentage than the 5.7% increase in 2019
(Figure 3.2.2), though the number of companies
funded decreased for the third year in a row(Figure 3.2.3). While there was a record high of
more than USD 40 billion in private investment
in 2020, that represents only a 9.3% increase
from 2019—compared with the largest increaseof 59.0%, observed between 2017 and 2018.
Moreover, the number of funded AI startups
continued a sharp decline from its 2017 peak.

Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

2015

Global Trend

PRIVATE INVESTMENT in FUNDED AI COMPANIES, 2015-20

Total Investment (in Millions of U.S. Dollars)

The following section analyzed the trend ofprivate investment in AI startups that havereceived investments of over USD 400,000 in
the last 10 years. While the amount of private
investment in AI has soared dramatically inrecent years, the rate of growth has slowed.

Figure 3.2.2

NUMBER OF NEWLY FUNDED AI COMPANIES in the WORLD, 2015-20
Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

Number of Companies

4,000

3,000

2,000

1,000

0
2015

2016

2017

2018

2019

2020
Figure 3.2.3

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

94

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 2 I N V E S T M E N T

Regional Comparison
As shown in Figure 3.2.4, the United States remains the leading destination for private investment, with over USD 23.6
billion in funding in 2020, followed by China (USD 9.9 billion) and the United Kingdom (USD 1.9 billion).
PRIVATE INVESTMENT in AI by COUNTRY, 2020
Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

United States
China
United Kingdom
IsraelCanadaGermany
FranceIndia
Japan
Singapore
Australia
0

5,000

10,00015,00020,00025,000Total Investment (in Millions of U.S. Dollars)
Figure 3.2.4

A closer examination of the three contenders leading
the AI race—the United States, China, and the European
Union—further validates the United States’ dominantposition in private AI investment. While China saw an
exceptionally high amount of private AI investment in

2018, its investment level in 2020 is less than half that of
the United States (Figure 3.2.5). It is important to note,however, that China has strong public investments in
AI. Both the central and local governments in China are
spending heavily on AI R&D.4

4 See “A Brief Examination of Chinese Government Expenditures on Artificial Intelligence R&D” (2020) by the Institute for Defense Analyses for more details.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

95

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 2 I N V E S T M E N T

PRIVATE INVESTMENT in AI by GEOGRAPHIC AREA, 2015-20
Source: CAPIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report
23,597
US

Total Investment (in Millions of U.S. Dollars)

25,00020,00015,000

9,933
China

10,000

6,662
Rest of the World

5,000
2,044
EU

0
2015

2016

2017

2018

2019

2020
Figure 3.2.5

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

96Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 2 I N V E S T M E N T

Focus Area Analysis
Figure 3.2.6 shows the ranking of the top 10 focus areasthat receive the greatest amount of private investment
in 2020 as well as their respective investment amountin 2019. The “Drugs, Cancer, Molecular, Drug Discovery”
area tops the list, with more than USD 13.8 billion in
private AI investment—4.5 times higher than 2019—
followed by “Autonomous Vehicles, Fleet, Autonomous
Driving, Road” (USD 4.5 billion), and “Students, Courses,
Edtech, English Language” (USD 4.1 billion).

In addition to Drugs, Cancer, Molecular, Drug Discovery,”both “Games, Fans, Gaming, Football” and “Students,
Courses, Edtech, English Language” saw a significant
increase in the amount of private AI investment from2019 to 2020. The former is largely driven by several
financing rounds to gaming and sports startups in
the United States and South Korea, while the latter isboosted by investments in an online education platform
in China.

GLOBAL PRIVATE INVESTMENT in AI by FOCUS AREA, 2019 vs 2020
Source: CapIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

Drugs, Cancer, Molecular, Drug Discovery
Autonomous Vehicles, Fleet, Autonomous
driving, Road
Students, Courses, Edtech, English language
Open Source, Compute, Hadoop, Devops
Speech Recognition, Computer interaction,
Dialogue, Machine translation
Money Laundering, Anti-fraud, Fraud Detection,
Fraud Prevention
Fashion, Shopping Experience, Beauty, Visual
SearchGames, Fans, Gaming, Football
Semiconductor, Chip, Data Centers, Processor
2019
2020

Bank, Card, Credit Cards, Gift

0

5,000

10,00015,000Total Investment (in Millions of U.S. Dollars)
Figure 3.2.6

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

97Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T YThis section reviews how corporations have capitalized on the advances in AI, using AI and automation to their advantage
and generating value at scale. While the number of corporations starting to deploy AI technologies has surged in recent
years, the economic turmoil and impact of COVID-19 in 2020 have slowed that rate of adoption. The latest trends incorporate AI activities are examined through data on the adoption of AI capabilities by McKinsey’s Global Survey on
AI, trends in robot installations across the globe by the International Federation of Robotics (IFR), and mentions of AI incorporate earnings calls by Prattle.

3.3 CORPORATE ACTIVITYI N D U S T RY A D O P T I O NGlobal Adoption of AI

This section shares the results of a McKinsey & Company
survey of 2,395 respondents: individuals representing
companies from a range of regions, industries, sizes,
functional specialties, and tenures.

The 2020 survey results suggest no increase in AI
adoption relative to 2019. Over 50% of respondents
say that their organizations have adopted AI in at least
one business function (Figure 3.3.1). In 2019, 58% of
respondents said their companies adopted AI in at least
one function, although the 2019 survey asked aboutcompanies’ AI adoption differently.

McKinsey & Company’s “The State of AI in 2020”
report contains the full results of this survey, including
insights on how different companies have adopted AI
across functions, core best practices shared among thecompanies that are generating the greatest value fromAI, and the impacts of the COVID-19 pandemic on thesecompanies’ AI investment plans.

In 2020, companies in developed Asia-Pacific countries
led in AI adoptions, followed by those in India and North
America. While AI adoption was about equal acrossregions in 2019, this year’s respondents working forcompanies in Latin America and in other developing
countries are much less likely to report adopting AI in atleast one business function.

AI ADOPTION by ORGANIZATIONS GLOBALLY, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report

Developed Asia-Pacific
India
All Geographies

North America
EuropeDeveloping Markets (incl.
China, MENA)
Latin America
0%

10%

20%

30%

40%

50%

60%

% of Respondents
Figure 3.3.1

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

98

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T Y

AI Adoption by Industry and Function

Type of AI Capabilities Adopted

Respondents representing companies in high tech andtelecom were most likely to report AI adoption in 2020,
similar to the 2019 results, followed in second place byboth financial services and automotive and assembly(Figure 3.3.2).

By industry, the type of AI capabilities adopted varies(Figure 3.3.4). Across industries, companies in 2020
were most likely to identify other machine learningtechniques, robotic process automation, and computer
vision as capabilities adopted in at least one business
function.

In another repeat from 2019 (and 2018), the 2020 survey
suggests that the functions where companies are most
likely to adopt AI vary by industry (Figure 3.3.3). For
example, respondents in the automotive and assembly
industry report greater AI adoption for manufacturingrelated tasks than any other; respondents in financial
services report greater AI adoption for risk functions; andrespondents in high tech and telecom report greater AI
adoption for product and service development functions.
Across industries, companies in 2020 are most likely
to report using AI for service operations (such as field
services, customer care, back office), product and service
development, and marketing and sales, similar to thesurvey results in 2019.

Industries tend to adopt AI capabilities that best serve
their core functions. For example, physical robotics,
as well as autonomous vehicles, are most frequently
adopted by industries where manufacturing anddistribution play a large role—such as automotive
and assembly, and consumer goods and retail.
Natural language processing capabilities, such as
text understanding, speech understanding, and text
generation, are frequently adopted by industries withhigh volumes of customer or operational data in text
forms; these include business, legal, and professional
services, financial services, healthcare, and high techand telecom.

AI ADOPTION by INDUSTRY, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index ReportHigh tech/Telecom
Automotive and Assembly
Financial Services
Business, Legal, andProfessional Services
Healthcare/Pharma
Consumer Goods/Retail
0%

10%

20%

30%

40%

50%

60%

70%

% of Respondents

Figure 3.3.2

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

99

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T Y

AI ADOPTION by INDUSTRY & FUNCTION, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report
Human
Resources

Industry

Manufacturing

Marketing And
Sales

Product and/or
Service
Development

Service
Operations

Risk

Strategy andCorporate
Finance

Supply-Chain
Management

All Industries

8%

12%

15%

21%

10%

21%

7%

9%

Automotive and Assembly

13%

29%

10%

21%

2%

16%

8%

18%

Business, Legal, and Professional
Services

13%

9%

16%

21%

13%

20%

10%

9%

Consumer Goods/Retail

1%

19%

20%

14%

3%

10%

2%

10%

Financial Services

5%

5%

21%

15%

32%

34%

7%

2%

Healthcare/Pharma

3%

12%

16%

15%

4%

11%

2%

6%

High Tech/Telecom

14%

11%

26%

37%

14%

39%

9%

12%

% of Respondents
Figure 3.3.3

AI CAPABILITIES EMBEDDED in STANDARD BUSINESS PROCESSES, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report

Deep
Learning

NL
NL Speech U NL Text Unde
Generation nderstanding rstanding

Other
MachineLearning
Techniques

PhysicalRobotics

Robotic
Process
Automation

Industry

Autonomous
VehiclesComputer
VisionConversation
al Interfaces

All Industries

7%

18%

15%

16%

11%

12%

13%

23%

13%

22%

Automotive and Assembly

20%

33%

16%

19%

12%

14%

19%

27%

31%

33%

Business, Legal, and Professional
Services

7%

13%

17%

19%

14%

15%

18%

25%

11%

13%

Consumer Goods/Retail

13%

10%

9%

6%

6%

6%

9%

12%

23%

14%

Financial Services

6%

18%

24%

19%

18%

19%

26%

32%

8%

37%

Healthcare/Pharma

1%

15%

10%

14%

12%

11%

15%

19%

10%

18%

High Tech/Telecom

9%

34%

32%

30%

18%

25%

33%

37%

14%

34%

% of Respondents
Figure 3.3.4

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

100

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T YConsideration and Mitigation of Risks from
Adopting AI
Only a minority of companies acknowledge the risks
associated with AI, and even fewer report taking steps
to mitigate those risks (Figure 3.3.5 and Figure 3.3.6).
Relative to 2019, the share of survey respondents
citing each risk as relevant has largely remained flat;that is, most changes were not statistically significant.
Cybersecurity remains the only risk a majority of
respondents say their organizations consider relevant.
A number of less commonly cited risks, such as national
security and political stability, were more likely to be
seen as relevant by companies in 2020 than in 2019.
Despite growing calls to attend to ethical concerns
associated with the use of AI, efforts to address theseconcerns in the industry are limited. For example,
concerns such as equity and fairness in AI use continue
to receive comparatively little attention from companies.
Moreover, fewer companies in 2020 view personal orindividual privacy as a risk from adopting AI comparedwith in 2019, and there is no change in the percentageof respondents whose companies are taking steps tomitigate this particular risk.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

Relative to 2019, the shareof survey respondents
citing each risk as
relevant has largely
remained flat; that is,most changes were notstatistically significant.
Cybersecurity remains
the only risk a majorityof respondents say theirorganizations consider
relevant.

101

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T Y

RISKS from ADOPTING AI THAT ORGANIZATIONS CONSIDER RELEVANT, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report

Cybersecurity
Regulatory
Compliance
Explainability
Personal/Individual
PrivacyOrganizational
Reputation
Workforce/Labor
Displacement
Equity And Fairness
Physical Safety
2019

National Security

2020

Political Stability
0%

10%

20%

30%

40%

50%

60%

% of Respondents
Figure 3.3.5

RISKS from ADOPTING AI THAT ORGANIZATIONS TAKE STEPS to MITGATE, 2020
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report

Cybersecurity
Regulatory
Compliance
Personal/Individual
Privacy
Explainability
Workforce/Labor
Displacement
Organizational
Reputation
Physical Safety
Equity And Fairness
National Security

2019

Political Stability

2020
0%

10%

20%

30%

40%

50%

% of Respondents
Figure 3.3.6

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

102

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

The Effect of COVID-19

3. 3 C O R P O R AT E
AC T I V I T Y

Despite the economic
downturn caused by
the pandemic, half of
respondents said the
pandemic had no effecton their investment inAI, while 27% actuallyreported increasing
their investment.

Despite the economic downturn caused by the pandemic,
half of respondents said the pandemic had no effecton their investment in AI, while 27% actually reported
increasing their investment. Less than a fourth ofbusinesses decreased their investment in AI (Figure
3.3.7).5 By industry, respondents in healthcare andpharma as well as automotive and assembly were themost likely to report that their companies had increased
investment in AI.

CHANGES in AI INVESTMENTS AMID the COVID-19 PANDEMIC
Source: McKinsey & Company, 2020 | Chart: 2021 AI Index Report

All Industries

23%

Automotive and Assembly

22%

Business, Legal, andProfessional Services

50%

27%

36%

42%

27%

48%

25%

Consumer Goods/Retail

23%

50%

26%

Financial Services

22%

50%

28%

11%

Healthcare/Pharma

44%

44%

23%

High Tech/Telecom
0%10%Decreased

52%
20%

30%
No Effect

40%

50%

24%
60%

70%

80%

90%

100%

Increased
Figure 3.3.7

5 Figures may not sum to 100% because of rounding.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

103

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T Y

INDUSTRIAL ROBOT INSTALLATIONS

Global Trend

Right now, AI is being deployed widely onto consumer
devices like smartphones and personal vehicles(e.g., self-driving technology). But relatively little AI
is deployed on actual robots.6 That may change as
researchers develop software to integrate AI-based
approaches with contemporary robots. For now, it ispossible to measure global sales of industrial robots
to draw conclusions about the amount of AI-readyinfrastructure being bought worldwide. While the COVID19-induced economic crisis will lead to a decline in robot
sales in the short term, the International Federation of
Robotics (IFR) expects the pandemic to generate global
growth opportunities for the robotics industry in themedium term.

After six years of growth, the number of new industrial
robots installed worldwide decreased by 12%, from422,271 units in 2018 to 373,240 units in 2019 (Figure
3.3.8). The decline is a product of trade tensions between
the United States and China as well as challenges facedby the two primary customer industries: automotive andelectrical/electronics.
With the automotive industry taking the lead (28% of
total installations), followed by electrical/electronics(24%), metal and machinery (12%), plastics and chemical
products (5%), and food and beverages (3%).7 It is
important to note that these metrics are a measurementof installed infrastructure that is susceptible to adopting
new AI technologies and does not indicate whether every
new robot used a significant amount of AI.

GLOBAL INDUSTRIAL ROBOT INSTALLATIONS, 2012-19
Source: International Federation of Robotics, 2020 | Chart: 2021 AI Index Report

Number of Units (in Thousands)

400

300

200

100

0
2012

2013

2014

2015

2016

2017

2018

2019
Figure 3.3.8

6 For more insights on the adoption of AI and robots by the industry, read the National Bureau of Economic Research working paper based on the 2018 Annual Business Survey by the U.S. Census Bureau,“Advancing Technologies Adoption and Use by U.S. Firms: Evidence From the Annual Business Survey” (2020).
7 Note that there is no information on the customer industry for approximately 20% of robots installed.

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

104

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

Regional Comparison

3. 3 C O R P O R AT E
AC T I V I T Y

Asia, Europe, and North
America—three of thelargest industrial robot
markets—all witnessed
the end of a six-year
growth period in robot
installations. North
America experienced the
sharpest decline, of 16%, in
2019, compared with 5% in
Europe and 13% in Asia.

Asia, Europe, and North America—three of the largest
industrial robot markets—all witnessed the end of a sixyear growth period in robot installations (Figure 3.3.9).
North America experienced the sharpest decline, of 16%,
in 2019, compared with 5% in Europe and 13% in Asia.
Figure 3.3.10 shows the number of installations in thefive major markets for industrial robot markets. All five—accounting for 73% of global robot installations—saw
roughly the same decline, except for Germany, which saw
a slight bump in installations between 2018 and 2019.
Despite the downward trend in China, it is worth notingthat the country had more industrial robots in 2019 thanthe other four countries combined.

NEW INDUSTRIAL ROBOT INSTALLATIONS by REGION, 2017-19
Source: International Federation of Robotics, 2020 | Chart: 2021 AI Index Report

Asia

EuropeNorth Amercia

OthersSouth & Central America

2017
2018
2019

Africa
0

100

200

300

Number of Units (in Thousands)
Figure 3.3.9

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

105

Artificial IntelligenceIndex Report 2021

CHAPTER 3:
THE ECONOMY

3. 3 C O R P O R AT E
AC T I V I T Y

NEW INDUSTRIAL ROBOT INSTALLATIONS in FIVE MAJOR MARKETS, 2017-19
Source: International Federation of Robotics, 2020 | Chart: 2021 AI Index Report

China

Japan

United States

2017
2018

South Korea

2019

Germany
Rest of the
World
0

50

100

150

200

Number of Units (in Thousands)
Figure 3.3.10

E A R N I N G S CA L L S
Mentions of AI in corporate earnings calls have increased substantially since 2013, as Figure 3.3.11 shows. In 2020, the numberof mentions of AI in earning calls was two times higher than mentions of big data, cloud, and machine learning combined,though that figure declined by 8.5% from 2019. The mentions of big data peaked in 2017 and have since declined by 57%.
MENTIONS of AI in CORPORATE EARNINGS CALLS, 2011-20
Source: Prattle & Liquidnet, 2020 | Chart: 2021 AI Index Report

5,000
4,734 Arti cial Intelligence

Number of AI Mentions

4,000

3,000

2,000
1,356 Maching Learning

1,000
652 Big Data
310 Cloud

0
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 3.3.11

CHAPTER 3 PREVIEWTA B L E O F C O N T E N T S

106

Artificial IntelligenceIndex Report 2021

CHAPTER 4:

AI Education

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

107

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

CHAPTER 4:

Chapter Preview
Overview

109

New CS PhDs in the United States

114

Chapter Highlights

110

New CS PhDs by Specialty

115

New CS PhDs with AI/ML andRobotics/Vision Specialties

117

4.1 STATE OF AI EDUCATION IN HIGHEREDUCATION INSTITUTIONS
111
Undergraduate AI Course Offerings

111

Undergraduate Courses
That Teach AI Skills

111

Intro-Level AI and ML Courses

111

Graduate AI Course Offerings

113

Graduate Courses That Focus
on AI Skills

113

Faculty Who Focus on AI Research

113

4.2 AI AND CS DEGREE GRADUATESIN NORTH AMERICA114

CS Undergraduate Graduates
in North America

114

New AI PhDs Employment
in North America

118

Industry vs. Academia

118

New International AI PhDs

119

4.3 AI EDUCATION IN THE
EUROPEAN UNION AND BEYOND

120

AI Offerings in EU27

120

By Content Taught in
AI-Related Courses

121International Comparison

122

HIGHLIGHT: AI BRAIN DRAIN
AND FACULTY DEPARTURE

123

ACCESS THE PUBLIC DATACHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

108

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O NOV E R V I E W

Overview
As AI has become a more significant driver of economic activity, there has been
increased interest from people who want to understand it and gain the necessary
qualifications to work in the field. At the same time, rising AI demands fromindustry are tempting more professors to leave academia for the private sector.
This chapter focuses on trends in the skills and training of AI talent through various
education platforms and institutions.
What follows is an examination of data from an AI Index survey on the state of AI
education in higher education institutions, along with a discussion on computerscience (CS) undergraduate graduates and PhD graduates who specialized in
AI-related disciplines, based on the annual Computing Research Association(CRA) Taulbee Survey. The final section explores trends in AI education in Europe,
drawing on statistics from the Joint Research Centre (JRC) at the European
Commission.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

109

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• An AI Index survey conducted in 2020 suggests that the world’s top universities haveincreased their investment in AI education over the past four years. The number of coursesthat teach students the skills necessary to build or deploy a practical AI model on theundergraduate and graduate levels has increased by 102.9% and 41.7%, respectively, in thelast four academic years.
• More AI PhD graduates in North America chose to work in industry in the past 10 years,
while fewer opted for jobs in academia, according to an annual survey from the Computing
Research Association (CRA). The share of new AI PhDs who chose industry jobs increased by
48% in the past decade, from 44.4% in 2010 to 65.7% in 2019. By contrast, the share of new AI
PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019.
• In the last 10 years, AI-related PhDs have gone from 14.2% of the total of CS PhDs granted in
the United States, to around 23% as of 2019, according to the CRA survey. At the same time,other previously popular CS PhDs have declined in popularity, including networking, software
engineering, and programming languages. Compilers all saw a reduction in PhDs granted
relative to 2010, while AI and Robotics/Vision specializations saw a substantial increase.
• After a two-year increase, the number of AI faculty departures from universities to industry
jobs in North America dropped from 42 in 2018 to 33 in 2019 (28 of these are tenured faculty
and five are untenured). Carnegie Mellon University had the largest number of AI faculty
departures between 2004 and 2019 (16), followed by the Georgia Institute of Technology (14)and University of Washington (12).
• The percentage of international students among new AI PhDs in North America continued torise in 2019, to 64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in
the United States and 8.6% have taken jobs outside the United States.
• In the European Union, the vast majority of specialized AI academic offerings are taught at
the master’s level; robotics and automation is by far the most frequently taught course in thespecialized bachelor’s and master’s programs, while machine learning (ML) dominates in thespecialized short courses.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

110

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4 .1 S TAT E O F A I E D U CAT I O N
I N H I G H E R E D U CAT I O N
INSTITUTIONS

4.1 STATE OF AI EDUCATION IN
HIGHER EDUCATION INSTITUTIONSNUMBER of UNDERGRADUATE COURSES THAT TEACH STUDENTS
the SKILLS NECESSARY to BUILD or DEPLOY a PRACTICAL AI
MODEL, AY 2016-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report

200

Number of Courses

In 2020, AI Index developed a survey that
asked computer science departments or
schools of computing and informatics
at top-ranking universities aroundthe world and in emerging economies
about four aspects of their AI education:
undergraduate program offerings, graduate
program offerings, offerings on AI ethics,
and faculty expertise and diversity. The
survey was completed by 18 universities
from nine countries.1 Results from the AI
Index survey indicate that universities haveincreased both the number of AI courses
they offer that teach students how to buildand deploy a practical AI model and thenumber of AI-focused faculty.

150

100

50

0

U N D E R G R A D UAT E A I
COURSE OFFERINGS
Course offerings at the undergraduatelevel were examined by evaluating trends
in courses that teach students the skills
necessary to build or deploy a practical AI
model, intro-level AI and ML courses, andenrollment statistics.

Undergraduate Courses That
Teach AI Skills
The survey results suggest that CS
departments have invested heavily inpractical AI courses in the past four
academic years (AY).2 The number of

2016-172017-18

2018-192019-20
Figure 4.1.1

courses on offer that teach students the skills necessary to build or
deploy a practical AI model has increased by 102.9%, from 102 in AY
2016–17 to 207 in AY 2019–20, across 18 universities (Figure 4.1.1).

Intro-Level AI and ML Courses
The data shows that the number of students who enrolled in or
attempted to enroll in an Introduction to Artificial Intelligencecourse and Introduction to Machine Learning course has jumped byalmost 60% in the past four academic years (Figure 4.1.2).3
The slight drop in enrollment in the intro-level AI and ML courses
in AY 2019–20 is mostly driven by the decrease in the number ofcourse offerings at U.S. universities. Intro-level course enrollment

1 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are—Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
2 See here for a list of keywords on practical artificial intelligence models provided to the survey respondents. A course is defined as a set of classes that require a minimum of 2.5 class hours (including
lecture, lab, TA hours, etc.) per week for at least 10 weeks in total. Multiple courses with the same titles and numbers count as one course.
3 For universities that have a cap on course registration, the number of students who attempted to enroll in the intro-level AI and ML courses are included.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

111

Artificial IntelligenceIndex Report 2021

NUMBER of STUDENTS WHO ENROLLED or ATTEMPTED to ENROLLin INTRO to AI and INTRO to ML COURSES, AY 2016-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report

10,0008,000
Number of Students

in the European Union has gradually
increased by 165% in the past four
academic years, while such enrollment
in the United States has seen a clear
dip in growth in the last academic
year (Figure 4.1.3). Six of the eight
U.S. universities surveyed say that the
number of (attempted) enrollments forthe introductory AI and ML courses has
decreased within the last year. Some
universities cited students taking leaves
during the pandemic as the main causeof the drop; others mentioned structuralchanges in intro-level AI course offerings—
such as creating Intro to Data Science lastyear—that may have driven students awayfrom traditional intro to AI and ML courses.

4 .1 S TAT E O F A I E D U CAT I O N
I N H I G H E R E D U CAT I O N
INSTITUTIONSCHAPTER 4:A I E D U CAT I O N

6,000

4,000

2,000

0
2016-172017-18

2018-192019-20
Figure 4.1.2

PERCENTAGE CHANGE in the NUMBER of STUDENTS WHO ENROLLED or ATTEMPTED to ENROLL in INTRO to AI and
INTRO to ML COURSES by GEOGRAPHIC AREA, AY 2016-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report165.1%, EU

Percentage Change (2016-17=1)

150%

100%

50%

21.0%, US0%
2016-172017-18

2018-192019-20
Figure 4.1.3

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

112

Artificial IntelligenceIndex Report 2021

4 .1 S TAT E O F A I E D U CAT I O N
I N H I G H E R E D U CAT I O N
INSTITUTIONSCHAPTER 4:A I E D U CAT I O N

G R A D UAT E A I C O U R S E
OFFERINGSNUMBER of GRADUATE COURSES THAT TEACH STUDENTS the
SKILLS NECESSARY to BUILD or DEPLOY a PRACTICAL AI MODEL,AY 2016-20

The survey also looks at course offeringsat the graduate or advanced degree level,
specifically at graduate courses that teachstudents the skills necessary to build or
deploy a practical AI model.4

Source: AI Index, 2020 | Chart: 2021 AI Index Report

Graduate offerings that teach students
the skills required to build or deploy apractical AI model increased by 41.7% in
the last four academic years, from 151
courses in AY 2016–17 to 214 in AY 2019–20(Figure 4.1.4).

Number of Courses

Graduate Courses That Focus
on AI Skills

200

150

100

50

0

FAC U LT Y W H O F O C U S O N
AI RESEARCH

2017-18

2018-192019-20
Figure 4.1.4

NUMBER of TENURE-TRACK FACULTY WHO PRIMARILY FOCUS
THEIR RESEARCH on AI, AY 2016-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report

150

Number of Faculty

As shown in Figure 4.1.5, the numberof tenure-track faculty with a primaryresearch focus on AI at the surveyed
universities grew significantly over the
past four academic years, in keepingwith the rising demand for AI classes anddegree programs. The number of AIfocused faculty grew by 59.1%, from 105 in
AY 2016–17 to 167 in AY 2019–20.

2016-17100

50

0
2016-172017-18

2018-192019-20
Figure 4.1.5

4 See here for a list of keywords on practical artificial intelligence models provided to the survey respondents. A course is defined as a set of classes that require a minimum of 2.5 class hours (including
lecture, lab, TA hours, etc.) per week for at least 10 weeks in total. Multiple courses with the same titles and numbers count as one course.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

113

Artificial IntelligenceIndex Report 2021

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

CHAPTER 4:A I E D U CAT I O NThis section presents findings from the annual Taulbee Survey from the Computing Research Association (CRA). The annual
CRA survey documents trends in student enrollment, degree production, employment of graduates, and faculty salaries inacademic units in the United States and Canada that grant doctoral degrees in computer science (CS), computer engineering (CE), or information (I). Academic units include departments of computer science and computer engineering or, in somecases, colleges or schools of information or computing.

4.2 AI AND CS DEGREE GRADUATESIN NORTH AMERICAC S U N D E R G R A D UAT E G R A D UAT E S
I N N O R T H A M E R I CA

NEW CS PHDS IN THE
U N I T E D S TAT E S

Most AI-related courses in North America are a part of
the CS course offerings at the undergraduate level. The
number of new CS undergraduate graduates at doctoral
institutions in North America has grown steadily in the last
10 years (Figure 4.2.1). More than 28,000 undergraduatescompleted CS degrees in 2019, around three times higher
than the number in 2010.

The section examines the trend of CS PhD graduates in
the United States with a focus on those with AI-related
specialties.5 The CRA survey includes 20 specialties intotal, two of which are directly related to the field of AI,
including “artificial intelligence/machine learning” and“robotics/vision.”

NUMBER of NEW CS UNDERGRADUATE GRADUATES at DOCTORAL INSTITUTIONS in NORTH AMERICA, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

Number of New CS Undergraduate Graduates

30,00025,00020,00015,00010,000

5,000

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019
Figure 4.2.1

5 New CS PhDs in this section include PhD graduates from academic units (departments, colleges, or schools within universities) of computer science in the United States.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

114

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

N E W C S P H D S BY S P E C I A LT Y
Among all computer science PhD graduates in 2019, those
who specialized in artificial intelligence/machine learning(22.8%), theory and algorithms (8.0%), and robotics/
vision (7.3%) top the list (Figure 4.2.2). The AI/ML specialty
has been the most popular in the past decade, and thenumber of AI/ML graduates in 2019 is higher than thenumber of the next five specialties combined. Moreover,robotics/vision jumped from the eighth most popularspecialization in 2018 to the third in 2019.

Over the past 10 years, AI/ML and robotics/vision are the
CS PhD specializations that exhibit the most significant
growth, relative to 18 other specializations (Figure 4.2.3).
The percentage of AI/ML-specialized CS PhD graduates
among all new CS PhDs in 2020 is 8.6 percentage points(pp) larger than in 2010, followed by robotics/visionspecialized doctorates at 2.4 pp. By contrast, the shareof CS PhDs specializing in networks (-4.8 pp), software
engineering (-3.6 pp), and programming languages/
compilers (-3.0 pp) experienced negative growth in 2020.

NEW CS PHDS (% of TOTAL) in the UNITED STATES by SPECIALITY, 2019
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

Artificial Intelligence/Machine Learning
Theory and Algorithms
Robotics/VisionDatabases/Information Retrieval
Security/Information Assurance
Graphics/Visualization
Networks
Other
Software Engineering
Human-Computer Interaction
Operating Systems
Informatics: Biomedical/Other Science
Hardware/ArchitectureHigh Performance Computing
Programming Languages/Compilers
Social Computing/Social Informatics/CSCW
Computing Education
Scientific/Numerical Computing
Information Systems
Information Science
0%

5%

10%

15%

20%

% of New CS PhDs
Figure 4.2.2

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

115

Artificial IntelligenceIndex Report 2021

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

CHAPTER 4:A I E D U CAT I O NPERCENTAGE POINT CHANGE in NEW CS PHDS in the UNITED STATES from 2010 to 2019 by SPECIALTY
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

Artificial Intelligence/Machine Learning
Robotics/Vision
Human-Computer Interaction
Security/Information Assurance
Computing Education
Databases/Information Retrieval
High Performance Computing
Theory and Algorithms
Information Science
Social Computing/Social Informatics/CSCW
Information Systems
Operating Systems
Graphics/Visualization
Other
Hardware/Architecture
Informatics: Biomedical/Other Science
Scientific/Numerical Computing
Programming Languages/Compilers
Software Engineering
Networks
-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

Percentage Point Change in New CS PhDs
Figure 4.2.3

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

116

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

N E W C S P H D S W I T H A I / M L A N D R O B OT I C S/ V I S I O N S P E C I A LT I E S
Figure 4.2.4a and Figure 4.2.4b take a closer look at the number of recent AI PhDs specializing in AI/ML or robotics/
vision in the United States. Between 2010 and 2019, the number of AI/ML-focused graduates grew by 77%, while thepercentage of these new PhDs among all CS PhD graduates increased by 61%. The number of both AI/ML and robotics/
vision PhD graduates reached an all-time high in 2019.

NEW CS PHDS with AI/ML and ROBOTICS/VISIONSPECIALTY in the UNITED STATES, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

NEW CS PHDS (% of TOTAL) with AI/ML and
ROBOTICS/VISION SPECIALTY in the UNITED STATES,
2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

300

30%

New CS PhDs (% of Total)

Number of Graduates

22.8%

200

100

.

10%

7.3%

0%

0
2..

20%

2010 2011 2012 2013 2014 2015 2016 2017 2018 20192010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Number of Artificial Intelligence/Machine Learning New PhDs

Artificial Intelligence/Machine Learning Specialty (% of New CS PhDs)

Number of Robotics/Vision New PhDs

Robotics/Vision Specialty (% of New CS PhDs)

Figure 4.2.4a

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

Figure 4.2.4b

117

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

N E W A I P H D S E M P LOY M E N T I N N O R T H A M E R I CA
Where do new AI PhD graduates choose to work? This section captures the employment trends of new AI PhDs in
academia and industry across North America.6

Industry vs. Academia
In the past 10 years, the number of new AI PhD graduates in North America who chose industry jobs continues to grow, as
its share increased by 48%, from 44.4% in 2010 to 65.7% in 2019 (Figure 4.2.5a and Figure 4.2.5b). By contrast, the shareof new AI PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019. As is clear from Figure 4.2.5b,these changes are largely a reflection of the fact that the number of PhD graduates entering academia has remainedroughly level through the decade, while the large increase in PhD output is primarily being absorbed by the industry.

EMPLOYMENT of NEW AI PHDS to ACADEMIA or
INDUSTRY in NORTH AMERICA, 2010-19

EMPLOYMENT of NEW AI PHDS (% of TOTAL) to
ACADEMIA or INDUSTRY in NORTH AMERICA, 2010-19

Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report
65.7%

New AI PhDs (% of Total)

Number of New AI PhDs

60%
150

100

50

N.

0

40%

23.7%

20%

0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 20192010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Number of New AI PhDs to Academia

New AI PhDs to Academia (% of Total New AI PhDs)

Number of New AI PhDs to Industry

New AI PhDs to Industry (% of Total New AI PhDs)

100

Figure 4.2.5a

Figure 4.2.5b

6 New AI PhDs in this section include PhD graduates who specialize in artificial intelligence from academic units (departments, colleges, or schools within universities) of computer science, computerengineering, and information in the United States and Canada.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

118

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4.2 AI AND CS DEGREE
G R A D UAT E S I N N O R T H
A M E R I CA

NEW INTERNATIONAL AI PHDS (% of TOTAL NEW AI PHDS) in NORTH AMERICA, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

New International AI PhDs (% of Total New AI PhDs)

70%
64.3%

60%

50%

40%

30%

20%
2010

2011

INTERNATIONALNEW AI(% of TOTAL)inSTATES
2012
2013
2014
2015PHDS20162017
2018the UNITED
2019
by LOCATION OF EMPLOYMENT, 2019
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

N E W I N T E R N AT I O N A L
AI PHDS
The percentage of international students
among new AI PhD graduates in North
America continued to rise in 2019, to
64.3%—a 4.3 percentage point increase from2018 (Figure 4.2.6). For comparison, of all
PhDs with a known specialty area, 63.4% ofcomputer engineering, 59.6% of computerscience, and 29.5% of information recipientsare international students in 2019.
Moreover, among foreign AI PhD graduatesin 2019 in the United States specifically,
81.8% stayed in the United States for
employment and 8.6% have taken jobs
outside the United States (Figure 4.2.7).
In comparison, among all international
student graduates with known specialties,
77.9% have stayed in the United States
while 10.4% were employed elsewhere.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

Figure 4.2.6

INTERNATIONAL NEW AI PHDS (% of TOTAL) in the UNITED STATESby LOCATION OF EMPLOYMENT, 2019
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

9.6%
Unknown

9.6%
Unknown

8.6%
Outside the United States
8.6%
Outside the United States

81.8%
United States

81.8%
United States

Figure 4.2.7

119Artificial IntelligenceIndex Report 2021

4 . 3 A I E D U CAT I O N I N
THE EUROPEAN UNIONA N D B E YO N D

CHAPTER 4:A I E D U CAT I O NThis section presents research from the Joint Research Center at the European Commission that assessed the academic offeringsof advanced digital skills in 27 European Union member states as well as six other countries: the United Kingdom, Norway,
Switzerland, Canada, the United States, and Australia. This was the second such study,7 and the 2020 version addressed four
technological domains: artificial intelligence (AI), high performance computing (HPC), cybersecurity (CS), and data science (DS),applying text-mining and machine-learning techniques to extract content related to study programs addressing the specificdomains. See the reports “Academic Offer of Advanced Digital Skills in 2019–20. International Comparison” and “Estimation of
Supply and Demand of Tertiary Education Places in Advanced Digital Profiles in the EU,” for more detail.

4.3 AI EDUCATION IN THE
EUROPEAN UNION AND BEYOND
A I O F F E R I N G S I N E U 27
The study revealed a total number of 1,032 AI programs
across program scopes and program levels in the 27 EU
countries (Figure 4.3.1). The overwhelming majority of
specialized AI academic offerings in the EU are taught at
the master’s level, which leads to a degree that equips

students with strong competencies for the workforce.
Germany leads the other member nations in offering
the most specialized AI programs, followed by the
Netherlands, France, and Sweden. France tops the list inoffering the most AI programs at the master’s level.

NUMBER of SPECIALIZED AI PROGRAMS in EU27, 2019-20
Source: Joint Research Centre, European Commission, 2020 | Chart: 2021 AI Index Report

Number of Specialized AI Programs

Master30

Bachelor
Short Courses

20

10

Bulgaria

Luxembourg

Croatia

Malta

Slovenia

GreeceSlovakia

Cyprus

LatviaRomania

Czechia

Portugal

Austria

Lithuania

Estonia

Hungary

Belgium

Spain

PolandItaly

IrelandFinlandDenmark

SwedenFranceNetherlands

Germany

0

Figure 4.3.1

7 Note that the 2020 report introduced methodological improvements from the 2019 version; therefore, a strict comparison is not possible. Improvements include the removal of certain keywords andthe addition of others to identify the programs. Still, more than 90% of all detected programs in the 2020 edition are triggered by keywords present in the 2019 study.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

120

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

4 . 3 A I E D U CAT I O N I N
THE EUROPEAN UNIONA N D B E YO N D

By Content Taught in AI-Related Courses
What types of AI technologies are the most popularamong the course offerings in three levels of specialized
AI programs in the European Union? Data suggests thatrobotics and automation are by far the most frequently
taught courses in the specialized bachelor’s andmaster’s programs, while machine learning dominatesin the specialized short courses (Figure 4.3.2). As shortcourses cater to working professionals, the trend showsthat machine learning has become one of the key
competencies in the professional development andimplementation of AI.

It is also important to mention the role of AI ethics andAI applications, as both content areas claim a significant
share of the education offerings among the threeprogram levels. AI ethics—including courses on security,
safety, accountability, and explainability—accounts for
14% of the curriculum on average, while AI applications—such as courses on big data, the internet of things, andvirtual reality—take a similar share on average.

SPECIALIZED AI PROGRAMS (% of TOTAL) by CONTENT AREA in EU27, 2019-20
Source: Joint Research Centre, European Commission, 2020 | Chart: 2021 AI Index Report

Short Courses

Bachelor

Master

Robotics & Automation
Machine Learning
AI Applications
AI Ethics
Computer Vision
Natural Language Processing
Connected And Automated Vehicles
Knowledge Representation And Reasoning;
Planning; Searching; Optimization
Multi-Agent Systems
Philosophy Of AI
AI (Generic)Audio Processing
0%

20%

40%

0%

20%

40%

0%

20%

40%

% of Total Specialized AI Programs
Figure 4.3.2

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

121

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

I N T E R N AT I O N A L C O M PA R I S O N
The JRC report compared AI education in the 27 EU
member states with other countries in Europe, including
Norway, Switzerland, and the United Kingdom, as well
as Canada, the United States, and Australia. Figure 4.3.3
shows the total number of 1,680 specialized AI programsin all countries considered in the 2019–20 academic
year. The United States appears to have offered moreprograms specialized in AI than any other geographic
area, although EU27 comes in a close second in terms ofthe number of AI-specialized master’s programs.

4 . 3 A I E D U CAT I O N I N
THE EUROPEAN UNIONA N D B E YO N D

The United States appears
to have offered moreprograms specialized
in AI than any othergeographic area although
EU27 comes in a close
second in terms of thenumber of AI-specialized
master’s programs.

NUMBER of SPECIALIZED AI PROGRAMS by GEOGRAPHIC AREA and LEVEL, 2019-20
Source: Joint Research Centre, European Commission, 2020 | Chart: 2021 AI Index Report

United States
UnitedKingdom
EU27
Australia
CanadaBachelor
MasterNorwayShort Courses

Switzerland
0

100

200

300

Number of Courses
Figure 4.3.3

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

122Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

HIGHLIGHT

AI Brain Drain and Faculty Departure
2018. They relied on data hand-collected from
LinkedIn as well as authors’ affiliation from the
Scopus database of academic publications andconferences to complement the results from theLinkedIn search.

Michael Gofman and Zhao Jin, researchers
from the University of Rochester and Cheung
Kong Graduate School of Business, respectively,
published a paper titled “Artificial Intelligence,
Education, and Entrepreneurship” in 2019 that
explores the relationship between domainspecific knowledge of university students andtheir ability to establish startups and attract
funding.8 For the source of variation in students’AI-specific knowledge, the co-authors used thedeparture of AI professors—what they referred
to as “an unprecedented brain drain”—fromuniversities to industry between 2004 andThe paper found that AI faculty departures
have a negative effect on AI startups foundedby students who graduate from universities
where those professors used to work, withthe researchers pointing to a chilling effect on
future AI entrepreneurs in the years following
the faculty departures. PhD students are themost affected, compared with undergraduate

NUMBER of AI FACULTY DEPARTURES in NORTH AMERICA, 2004-19
Source: Gofman and Jin, 2020 | Chart: 2021 AI Index Report

40

Number of Faculty

TenuredTotal Number of Departures

30

Untenured

20

10

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

0

Figure 4.4.1

8 See AI Brain Drain Index for more details.

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

123

Artificial IntelligenceIndex Report 2021

CHAPTER 4:A I E D U CAT I O N

HIGHLIGHT

AI Brain Drain and Faculty Departure (continued)
and master’s students, and the negative impact
intensifies when the AI professors who leave are
replaced by faculty from lower-ranked schoolsor untenured AI professors.

to industry dropped from 42 in 2018 to 33 in2019 (28 of these are tenured faculty and 5 areuntenured). Between 2004 and 2019, Carnegie
Mellon University had the largest number ofAI faculty departures in 2019 (16), followed by
the Georgia Institute of Technology (14) andUniversity of Washington (12), as shown in
Figure 4.4.2.

With the updated data of 2019 from Gofman
and Jin, Figure 4.4.1 shows that after a twoyear increase, the total number of AI faculty
departures from universities in North America

NUMBER of AI FACULTY DEPATURES in NORTH AMERICA (with UNIVERSITY AFFILIATION) by UNIVERSITY, 2004-18
Source: Gofman and Jin, 2020 | Chart: 2021 AI Index Report

Carnegie Mellon University
Georgia Institute of Technology
University of Washington
University of California, Berkeley
University of Toronto
Stanford University
University of Southern California
University of Texas at Austin
University of Michigan
University of Illinois at Urbana-Champaign
University of California, San Diego
Purdue University
Harvard University
0

2

4

6

8

10

12

14

16

Number of Faculty
Figure 4.4.2

CHAPTER 4 PREVIEWTA B L E O F C O N T E N T S

124

Artificial IntelligenceIndex Report 2021

CHAPTER 5:

Ethical Challengesof AI Applications

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

125

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

Artificial IntelligenceIndex Report 2021

CHAPTER 5:

Chapter Preview
Overview

127

Chapter Highlights

128

5.1 AI PRINCIPLES AND FRAMEWORKS 129
5.2 GLOBAL NEWS MEDIA

131

5.3 ETHICS AT AI CONFERENCES

132

5.4 ETHICS OFFERINGS AT HIGHEREDUCATION INSTITUTIONS134

ACCESS THE PUBLIC DATACHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

126

Artificial IntelligenceIndex Report 2021

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N SOV E R V I E W

Overview
As artificial intelligence–powered innovations become ever more prevalent in ourlives, the ethical challenges of AI applications are increasingly evident and subject
to scrutiny. As previous chapters have addressed, the use of various AI technologies
can lead to unintended but harmful consequences, such as privacy intrusion;
discrimination based on gender, race/ethnicity, sexual orientation, or gender
identity; and opaque decision-making, among other issues. Addressing existing
ethical challenges and building responsible, fair AI innovations before they get
deployed has never been more important.
This chapter tackles the efforts to address the ethical issues that have arisenalongside the rise of AI applications. It first looks at the recent proliferation of
documents charting AI principles and frameworks, as well as how the media covers
AI-related ethical issues. It then follows with a review of ethics-related researchpresented at AI conferences and what kind of ethics courses are being offered bycomputer science (CS) departments at universities around the world.
The AI Index team was surprised to discover how little data there is on this topic.
Though a number of groups are producing a range of qualitative or normative
outputs in the AI ethics domain, the field generally lacks benchmarks that can beused to measure or assess the relationship between broader societal discussions
about technology development and the development of the technology itself.
One datapoint, covered in the technical performance chapter, is the study by the
National Institute of Standards and Technology on facial recognition performancewith a focus on bias. Figuring out how to create more quantitative data presentsa challenge for the research community, but it is a useful one to focus on.
Policymakers are keenly aware of ethical concerns pertaining to AI, but it is easierfor them to manage what they can measure, so finding ways to translate qualitativearguments into quantitative data is an essential step in the process.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

127

Artificial IntelligenceIndex Report 2021

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• The number of papers with ethics-related keywords in titles submitted to AI conferences
has grown since 2015, though the average number of paper titles matching ethicsrelated keywords at major AI conferences remains low over the years.
• The five news topics that got the most attention in 2020 related to the ethical use of AI
were the release of the European Commission’s white paper on AI, Google’s dismissal of
ethics researcher Timnit Gebru, the AI ethics committee formed by the United Nations,
the Vatican’s AI ethics plan, and IBM’s exiting the facial-recognition businesses.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

128

Artificial IntelligenceIndex Report 2021

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

5.1 A I P R I N C I P L E S
A N D F R A M E WO R KS

5.1 AI PRINCIPLES AND FRAMEWORKS
Since 2015, governments, private companies,
intergovernmental organizations, and research/
professional organizations have been producing
normative documents that chart the approaches tomanage the ethical challenges of AI applications.
Those documents, which include principles, guidelines,
and more, provide frameworks for addressing theconcerns and assessing the strategies attached todeveloping, deploying, and governing AI within variousorganizations. Some common themes that emerge fromthese AI principles and frameworks include privacy,
accountability, transparency, and explainability.
The publication of AI principles signals that organizations
are paying heed to and establishing a vision for AI
governance. Even so, the proliferation of so-called ethicalprinciples has met with criticism from ethics researchersand human rights practitioners who oppose theimprecise usage of ethics-related terms. The critics also
point out that they lack institutional frameworks andare non-binding in most cases. The vague and abstract
nature of those principles fails to offer direction on how
to implement AI-related ethics guidelines.
Researchers from the AI Ethics Lab in Boston created a
ToolBox that tracks the growing body of AI principles.
A total of 117 documents relating to AI principles were
published between 2015 and 2020. Data shows that
research and professional organizations were among theearliest to roll out AI principle documents, and privatecompanies have to date issued the largest number ofpublications on AI principles among all organization types(Figure 5.1.1). Europe and Central Asia have the highest
number of publications as of 2020 (52), followed by North
America (41), and East Asia and Pacific (14), according
to Figure 5.1.2. In terms of rolling out ethics principles,
2018 was the clear high-water mark for tech companies—
including IBM, Google, and Facebook—as well as various
U.K., EU, and Australian government agencies.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

Europe and Central
Asia have the highest
number of publications
as of 2020 (52), followedby North America(41), and East Asia andPacific (14). In termsof rolling out ethics
principles, 2018 was theclear high-water markfor tech companies—
including IBM, Google,
and Facebook—as wellas various U.K., EU, andAustralian government
agencies.

129

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

Artificial IntelligenceIndex Report 2021

5.1 A I P R I N C I P L E S
A N D F R A M E WO R KS

NUMBER of NEW AI ETHICS PRINCIPLES by ORGANIZATION TYPE, 2015-20
Source: AI Ethics Lab, 2020 | Chart: 2021 AI Index Report45

Research/Professional Organization
Private Company

Number of New AI Ethics Principles

40

13Intergovernmental Organization/Agency
Government Agency

30

28
6
19

20

17

11

10

2

3

2

4
8

2

11

5

2015

2016

2017

15

9
4

2

0

23

2018

2019

2020
Figure 5.1.1

NUMBER of NEW AI ETHICS PRINCIPLES by REGION, 2015-20
Source: AI Ethics Lab, 2020 | Chart: 2021 AI Index Report45

East Asia & Pacific

Number of New AI Ethics Principles

6

Europe & Central Asia

40

GlobalLatin America & Caribbean
Middle East & North Africa

30

20

North America

285South Asia
20

17

12

23

13

5

10

16

2

2

2015

2016

9

7

7

0
2017

2018

2019

2020
Figure 5.1.2

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

130

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

Artificial IntelligenceIndex Report 2021

5. 2 G LO BA L
NEWS MEDIA

5.2 GLOBAL NEWS MEDIA
How has the news media covered the topic of the ethicaluse of AI technologies? This section analyzed datafrom NetBase Quid, which searches the archived news
database of LexisNexis for articles that discuss AI ethics1,
analyzing 60,000 English-language news sources andover 500,000 blogs in 2020.
The search found 3,047 articles related to AI technologies
that include terms such as “human rights,” “humanvalues,” “responsibility,” “human control,” “fairness,”“discrimination” or “nondiscrimination,” “transparency,”“explainability,” “safety and security,” “accountability,”and “privacy.” (See the Appendix for more details on
search terms.) NetBase Quid clustered the resulting
media narratives into seven large themes based on
language similarity.

Figure 5.2.1 shows that articles relating to AI ethics
guidance and frameworks topped the list of the most
covered news topics (21%) in 2020, followed by researchand education (20%), and facial recognition (20%).
The five news topics that received the most attention in2020 related to the ethical use of AI were:	1. The release of the European Commission’s
white paper on AI (5.9%)	2. Google’s dismissal of ethics researcher
Timnit Gebru (3.5%)
3. The AI ethics committee formed by theUnited Nations (2.7%)
4. The Vatican’s AI ethics plan (2.6%)
5. IBM exiting the facial-recognition businesses (2.5%).

NEWS COVERAGE on AI ETHICS (% of TOTAL) by THEME, 2020
Source: CAPIQ, Crunchbase, and NetBase Quid, 2020 | Chart: 2021 AI Index Report

Guidance, Framework
Research, Education
Facial Recognition
Algorithm Bias
Robots, Autonomous Cars
AI Explainability
Data Privacy
Enterprise Efforts
0%

5%

10%

15%

20%

% of Total News Coverage on AI Ethics
Figure 5.2.1

1 The methodology for this is looking for articles that contain keywords related to AI ethics as determined by a Harvard research study.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

131

Artificial IntelligenceIndex Report 2021

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

5. 3 E T H I C S AT
AI CONFERENCES

5.3 ETHICS AT AI CONFERENCES
Researchers are writing more papers that focus directlyon the ethics of AI, with submissions in this area morethan doubling from 2015 to 2020. To measure the roleof ethics in AI research, researchers from the Federal
University of Rio Grande do Sul in Porto Alegre, Brazil,
searched ethics-related terms in the titles of papers inleading AI, machine learning, and robotics conferences.
As Figure 5.3.1 shows, there has been a significant
increase in the number of papers with ethics-related
keywords in titles submitted to AI conferences since 2015.
Further analysis in Figure 5.3.2 shows the average
number of keyword matches throughout all publications
among the six major AI conferences. Despite the growing
mentions in the previous chart, the average number of
paper titles matching ethics-related keywords at major AI
conferences remains low over the years.
Changes are coming to AI conferences, though. Startingin 2020, the topic of ethics was more tightly integrated
into conference proceedings. For instance, the Neural
Information Processing Systems (NeurIPS) conference,
one of the biggest AI research conferences in theworld, asked researchers to submit “Broader Impacts”
statements alongside their work for the first time in 2020,
which led to a deeper integration of ethical concerns
into technical work. Additionally, there has been a
recent proliferation of conferences and workshopsthat specifically focus on responsible AI, including
the new Artificial Intelligence, Ethics, and Society
Conference by the Association for the Advancement of
Artificial Intelligence and the Conference on Fairness,
Accountability, and Transparency by the Association for
Computing Machinery.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

There has been asignificant increasein the number of
papers with ethicsrelated keywords intitles submitted to
AI conferences since2015. Further analysis
shows the average
number of keyword
matches throughout
all publications among
the six major AI
conferences.

132

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

Artificial IntelligenceIndex Report 2021

5. 3 E T H I C S AT
AI CONFERENCESNUMBER of PAPER TITLES MENTIONING ETHICS KEYWORDS at AI CONFERENCES, 2000-19
Source: Prates et al., 2018 | Chart: 2021 AI Index Report

80
70

Number of Paper Titles

60

40

20

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 5.3.1

AVERAGE NUMBER of PAPER TITLES MENTIONING ETHICS KEYWORDS at SELECT LARGE AI CONFERENCES,
2000-19
Source: Prates et al., 2018 | Chart: 2021 AI Index Report

Average Number of Keywords Matches

AAAI
ICML

0.06

ICRA
IJCAI
IROS
NIPS/NeurIPS

0.04

0.02

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.00

Figure 5.3.2

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

133

CHAPTER 5:
E T H I CA L C H A L L E N G E SO F A I A P P L I CAT I O N S

Artificial IntelligenceIndex Report 2021

5.4 E T H I C S O F F E R I N G S
AT H I G H E R E D U CAT I O N
INSTITUTIONS

5.4 ETHICS OFFERINGS ATHIGHER EDUCATION INSTITUTIONS
Chapter 4 introduced a survey of computer science
departments or schools at top universities around theworld in order to assess the state of AI education inhigher education institutions.2 In part, the survey asked
whether the CS department or university offers theopportunity to learn about the ethical side of AI and CS.
Among the 16 universities that completed the survey, 13
reported some type of relevant offering.
Figure 5.4.1 shows that 11 of the 18 departments reporthosting keynote events or panel discussions on AI ethics,
while 7 of them offer stand-alone courses on AI ethics
in CS or other departments at their university. Some
universities also offer classes on ethics in the computerscience field in general, including stand-alone CS
ethics courses or ethics modules embedded in the CS
curriculum offering.3

11 of the 18 departments
report hosting keynote
events or panel discussions
on AI ethics, while 7 ofthem offer stand-alone
courses on AI ethics in CSor other departments attheir university.

AI ETHICS OFFERING at CS DEPARTMENTS of TOP UNIVERSITIES around the WORLD, AY 2019-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report

Keynote events or panel discussions on AI ethics
University-wide undergraduate general requirement
courses on broadly defined ethics
Standalone courses on AI ethics in CS or Other
Departments
Stand-alone courses on CS ethics in CS or Other
Departments
Ethics modules embedded into CS courses
AI ethics–related student groups/organizations
0

2

4

6

8

10

12
Figure 5.4.1

2 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are—Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
3 The survey did not explicitly present “Ethics modules embedded into CS courses” as an option. Selections were filled in the “Others” option. This will be included in next year’s survey.

CHAPTER 5 PREVIEWTA B L E O F C O N T E N T S

134

Artificial IntelligenceIndex Report 2021

CHAPTER 6:

Diversity in AI

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

135

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

CHAPTER 6:

Chapter Preview
Overview

137

Chapter Highlights

138

6.1 GENDER DIVERSITY IN AI

139

Women in Academic AI Settings

139

Women in the AI Workforce

140

Women in Machine Learning Workshops 141
Workshop Participants

141

Demographics Breakdown142

6.2 RACIAL AND ETHNIC
DIVERSITY IN AI

144

New AI PhDs in the United Statesby Race/Ethnicity

144

New Computing PhDs in the
United States by Race/Ethnicity

145

CS Tenure-Track Facultyby Race/Ethnicity

146

Black in AI

146

6.3 GENDER IDENTITY ANDSEXUAL ORIENTATION IN AI

147

Queer in AI

147

Demographics Breakdown

147

Experience as Queer Practitioners

149

ACCESS THE PUBLIC DATACHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

136

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AIOV E R V I E W

Overview
While artificial intelligence (AI) systems have the potential to dramatically affect society,the people building AI systems are not representative of the people those systems aremeant to serve. The AI workforce remains predominantly male and lacking in diversity inboth academia and the industry, despite many years highlighting the disadvantages andrisks this engenders. The lack of diversity in race and ethnicity, gender identity, and sexual
orientation not only risks creating an uneven distribution of power in the workforce, butalso, equally important, reinforces existing inequalities generated by AI systems, reduces
the scope of individuals and organizations for whom these systems work, and contributes
to unjust outcomes.
This chapter presents diversity statistics within the AI workforce and academia. It draws
on collaborations with various organizations—in particular, Women in Machine Learning(WiML), Black in AI (BAI), and Queer in AI (QAI)— each of which aims to improve diversity
in some dimension in the field. The data is neither comprehensive nor conclusive. In
preparing this chapter, the AI Index team encountered significant challenges as a resultof the sparsity of publicly available demographic data. The lack of publicly available
demographic data limits the degree to which statistical analyses can assess the impactof the lack of diversity in the AI workforce on society as well as broader technology
development. The diversity issue in AI is well known, and making more data available
from both academia and industry is essential to measuring the scale of the problem andaddressing it.
There are many dimensions of diversity that this chapter does not cover, including AI
professionals with disabilities; nor does it consider diversity through an intersectional
lens. Other dimensions will be addressed in future iterations of this report. Moreover,these diversity statistics tell only part of the story. The daily challenges of minorities andmarginalized groups working in AI, as well as the structural problems within organizations
that contribute to the lack of diversity, require more extensive data collection and analysis.

1 We thank Women in Machine Learning, Black in AI, and Queer in AI for their work to increase diversity in AI, for sharing their data, and for partnering with us.

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

137

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• The percentages of female AI PhD graduates and tenure-track computer science(CS) faculty have remained low for more than a decade. Female graduates of AI PhD
programs in North America have accounted for less than 18% of all PhD graduates on
average, according to an annual survey from the Computing Research Association(CRA). An AI Index survey suggests that female faculty make up just 16% of all tenuretrack CS faculty at several universities around the world.
• The CRA survey suggests that in 2019, among new U.S. resident AI PhD graduates,
45% were white, while 22.4% were Asian, 3.2% were Hispanic, and 2.4% were African
American.
• The percentage of white (non-Hispanic) new computing PhDs has changed little
over the last 10 years, accounting for 62.7% on average. The share of Black or
African American (non-Hispanic) and Hispanic computing PhDs in the same period issignificantly lower, with an average of 3.1% and 3.3%, respectively.
• The participation in Black in AI workshops, which are co-located with the Conference
on Neural Information Processing Systems (NeurIPS), has grown significantly in recent
years. The numbers of attendees and submitted papers in 2019 are 2.6 times higher thanin 2017, while the number of accepted papers is 2.1 times higher.
• In a membership survey by Queer in AI in 2020, almost half the respondents said they
view the lack of inclusiveness in the field as an obstacle they have faced in becoming
a queer practitioner in the AI/ML field. More than 40% of members surveyed said they
have experienced discrimination or harassment as a queer person at work or school.

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

138

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6.1 G E N D E R
DIVERSITY IN AI

6.1 GENDER DIVERSITY IN AI
WO M E N I N ACA D E M I C
AI SETTINGSTENURE-TRACK FACULTY at CS DEPARTMENTS of
TOP UNIVERSITIES
the
WORLD
by GENDER,TENURE-TRACKaround
FACULTY
at CS
DEPARTMENTS
of
Chapter 4 introduced the AI Index survey that evaluates
TOP
UNIVERSITIES
aroundthe
WORLD
byGENDER,AY 2019-20
the state of AI education in CS departments at top
AY 2019-20
Source: AI Index,
2020 | Chart: 2021 AI Index Report
Source: AI Index, 2020 | Chart: 2021 AI Index Report
universities around the world, along with the Computer
Research Association’s annual Taulbee Survey on the
110 (16.1%)
enrollment, production, and employment of PhDs
110
(16.1%)
FemaleFemalein information, computer science, and computerengineering in North America.
Data from both surveys show that the percentage offemale AI and CS PhD graduates as well as tenure-track CS
faculty remains low. Female graduates of AI PhD programs
and CS PhD programs have accounted for 18.3% of all
PhD graduates on average within the past 10 years (Figure
6.1.1). Among the 17 universities that completed the AI
Index survey of CS programs globally, female faculty makeup just 16.1% of all tenure-track faculty whose primaryresearch focus area is AI (Figure 6.1.2).

575 (83.9%)
Male575 (83.9%)
Male

Figure 6.1.2

FEMALE NEW AI and CS PHDS (% of TOTAL NEW AI and CS PHDS) in NORTH AMERICA, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

Female New AI PhDs (% of All New AI PhDs)

30%

25%
22.1% AI20%

20.3% CS

15%

10%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019
Figure 6.1.1

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

139

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6.1 G E N D E R
DIVERSITY IN AI

WO M E N I N T H E A I WO R K F O R C E
Chapter 3 introduced the “global relative AI skills
penetration rate,” a measure that reflects the prevalenceof AI skills across occupations, or the intensity with whichpeople in certain occupations use AI skills. Figure 6.1.3
shows AI skills penetration by country for female andmale labor pools in a set of select countries.2 The datasuggest that across the majority of these countries, theAI skills penetration rate for women is lower than thatfor men. Among the 12 countries we examined, India,
South Korea, Singapore, and Australia are the closest toreaching equity in terms of the AI skills penetration rateof females and males.

This data suggests thatacross the majority of
select countries, theAI skills penetration
rate for women is lower
than it is for men.

RELATIVE AI SKILLS PENETRATION RATE by GENDER, 2015-20
Source: LinkedIn, 2020 | Chart: 2021 AI Index Report

India
United States
South Korea
Singapore
China
CanadaFranceGermany
Australia
United Kingdom

FemaleMale

South Africa
Italy
0

1

2

3

Relative AI Skills Penetration Rate
Figure 6.1.3

2 Countries included are a select sample of eligible countries with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were included in this sample
because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full a picture
as other countries, and should be interpreted accordingly.

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

140

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6.1 G E N D E R
DIVERSITY IN AI

WO M E N I N M AC H I N E L E A R N I N G
WO R KS H O P S
Women in Machine Learning, founded in 2006 by
Hanna Wallach, Jenn Wortman, and Lisa Wainer, is anorganization that runs events and programs to support
women in the field of machine learning (ML). Thissection presents statistics from its annual technicalworkshops, which are held at NeurIPS. In 2020, WiML
also hosted for the first time a full-day “Un-Workshop” at
the International Conference on Machine Learning 2020,
which drew 812 participants.

Workshop Participants
The number of participants attending WiML workshops at
NeurIPS has been steadily increasing since the workshops
were first offered in 2006. According to the organization,
the WiML workshop in 2020 was completely virtual

because of the pandemic and delivered on a new platform(Gather.Town); these two factors may make attendance
numbers harder to compare to those of previous years.
Figure 6.1.4 shows an estimate of 925 attendees in 2020,
based on the number of individuals who accessed thevirtual platform.
In the past 10 years, WiML workshops have expanded
their programs to include mentoring roundtables, wheremore senior participants offer one-on-one feedback andprofessional advice, in addition to the main session that
includes keynotes and poster presentations. Similar
opportunities may have contributed to the increase inattendance since 2014. Between 2016 and 2019, the WiML
workshop attendance is on average about 10% of theoverall NeurIPS attendance.

NUMBER of PARTICIPANTS at WIML WORKSHOP at NEURIPS, 2006-20
Source: Women in Machine Learning, 2020 | Chart: 2021 AI Index Report

1,000
925

Number of Participants

800

600

400

200

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

0

Figure 6.1.4

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

141

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

Demographics Breakdown
The following geographic, professional position, andgender breakdowns are based only on participants atthe 2020 WiML workshop at NeurIPS who consented tohaving the information aggregated and who spent atleast 10 minutes on the virtual platform through which
the workshop was offered. Among the participants,
89.5% were women and/or nonbinary, 10.4% were men(Figure 6.1.5), and a large majority were from North
America (Figure 6.1.6). Further, as shown in Figure 6.1.7,
students—including PhD, master’s, and undergraduate
students—make up more than half the participants(54.6%). Among participants who work in the industry,
research scientist/engineer and data scientist/engineer
are the most commonly held professional positions.

PARTICIPANTS of WIML WORKSHOP at NEURIPS(% of
TOTAL) byofGENDER,
2020 at NEURIPS
PARTICIPANTS
WIML WORKSHOP
Source:in Machine
Learning, 2020
| Chart: 2021 AI Index Report(%Womenof TOTAL)
by GENDER,
2020
Source: Women in Machine Learning, 2020 | Chart: 2021 AI Index Report

6.1 G E N D E R
DIVERSITY IN AI

Among theparticipants, 89.5%
were women and/or nonbinary, 10.4%
were men, and a large
majority were from
North America. Further,students—including
PhD, master’s, andundergraduate
students—make up
more than half theparticipants (54.6%).

10.4%10.4%
Man Man

89.5%
Woman and/or nonbinary

89.5%
Woman and/or nonbinary
Figure 6.1.5

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

142

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6.1 G E N D E R
DIVERSITY IN AIPARTICIPANTS of WIML WORKSHOP at NEURIPS (% of TOTAL) by CONTINENT of RESIDENCE, 2020
Source: Women in Machine Learning, 2020 | Chart: 2021 AI Index Report

North America
EuropeAsia
AfricaCentral, South America, and the Carribean
Australia and Oceania
Middle East
0%

10%

20%

30%

40%

50%

60%

% of Participants
Figure 6.1.6

PARTICIPANTS of WIML WORKSHOP at NEURIPS (% of TOTAL) by TOP 10 PROFESSIONAL POSITIONS, 2020
Source: Women in Machine Learning, 2020 | Chart: 2021 AI Index Report

PhD Student
Research Scientist/Engineer
MSc Student
Data scientist/Engineer
Undergraduate Student
Postdoctoral Researcher
Software Engineer
Professor (Pre-Tenure)
Professor (Post-Tenure)
Program/Product Manager
0%

5%

10%

15%

20%

25%

30%

35%

% of Participants
Figure 6.1.7

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

143

Artificial IntelligenceIndex Report 2021

6. 2 R AC I A L
AND ETHNIC
DIVERSITY IN AI

CHAPTER 6:
DIVERSITY IN AI

6.2 RACIAL AND ETHNIC DIVERSITY IN AI
N E W A I P H D S I N T H E U N I T E D S TAT E S BY R AC E / E T H N I C I T Y
According to the CRA Taulbee Survey, among the new AI PhDs in 2019 who are U.S. residents, the largest
percentage (45.6%) are white (non-Hispanic), followed by Asian (22.4%). By comparison, 2.4% were African
American (non-Hispanic) and 3.2% were Hispanic (Figure 6.2.1).

NEW U.S. RESIDENT AI PHDS (% of TOTAL) by RACE/ETHNICITY, 2019

NEW U.S. RESIDENT
AICRA
PHDS
(% Survey,of TOTAL)
by RACE/ETHNICITY,
Source:Taulbee2020 | Chart:
2021 AI Index Report 2019
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

22.4% Asian
45.6% White(non-Hispanic)

22.4% Asian

45.6% White(non-Hispanic)
2.4% Black or African
American(non-Hispanic)
3.2%
Hispanic
1.6% Multiracial(non-Hispanic)

24.8% Unknown

TA B L E O F C O N T E N T S

3.2%
Hispanic
1.6% Multiracial(non-Hispanic)

24.8% Unknown

CHAPTER 6 PREVIEW

2.4% Black or African
American(non-Hispanic)

Figure 6.2.1

144

Artificial IntelligenceIndex Report 2021

6. 2 R AC I A L
AND ETHNIC
DIVERSITY IN AI

CHAPTER 6:
DIVERSITY IN AI

NEW COMPUTING PHDS IN
T H E U N I T E D S TAT E S BY
R AC E / E T H N I C I T Y

The CRA survey
indicates that thepercentage of white(non-Hispanic) new
PhDs has changed little
over the last 10 years,
accounting for 62.7%
on average.

Figure 6.2.2 shows all PhDs awarded in the United States
to U.S. residents across departments of computer science(CS), computer engineering (CE), and information (I)between 2010 and 2019. The CRA survey indicates thatthe percentage of white (non-Hispanic) new PhDs haschanged little over the last 10 years, accounting for 62.7%
on average. The share of new Black or African American(non-Hispanic) and Hispanic computing PhDs in the same
period is significantly lower, with an average of 3.1%
and 3.3%, respectively. We are not able to compare thenumbers between new AI and CS PhDs in 2019 becauseof the number of unknown cases (24.8% for new AI PhDs
and 8.5% for CS PhDs).

NEW COMPUTING PHDS, U.S. RESIDENT (% of TOTAL) by RACE/ETHNICITY, 2010-19
Source: CRA Taulbee Survey, 2020 | Chart: 2021 AI Index Report

70%

New Computing PhDs, U.S. Resident (% of Total)

60%
58.9% White (non-Hispanic)

50%

40%

30%
24.4% Asian

20%

10%

8.5% Unknown
3.2% Hispanic, any race
2.5% Black or African American (non-Hispanic)
1.7% Multiracial (non-Hispanic)

0%0.6% Native Hawaiian/Pac Islander
0.3% Amer Indian or Alaska Native

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 6.2.2

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

145

Artificial IntelligenceIndex Report 2021

C S T E N U R E -T R AC K
FAC U LT Y BY R AC E /ETHNICITY

6. 2 R AC I A L
AND ETHNIC
DIVERSITY IN AI

CHAPTER 6:
DIVERSITY IN AITENURE-TRACK FACULTY (% of TOTAL) at CS DEPARTMENTS
of TOP UNIVERSITIES in the WORLD by RACE/ETHNICITY, 2019-20
TENURE-TRACKFACULTY(% ofReport
TOTAL) at CS DEPARTMENTS
Source:AI Index, 2020 | Chart:
2021 AI Index

of TOP UNIVERSITIES in the WORLD by RACE/ETHNICITY, 2019-20
Source: AI Index, 2020 | Chart: 2021 AI Index Report

0.8%
Hispanic,
0.6% 0.8%
Latino, or
Hispanic,
Black
or
0.6%
Black orAfricanLatino, or Spanish
SpanishAfricanorigin14.3%
14.3%Asian

Figure 6.2.3 shows data from the AI Index
education survey.3 Among 15 universities
that completed the question pertaining
to the racial makeup of their faculty,
approximately 67.0% of the tenure-track
faculty are white, followed by Asian (14.3%),
other races (8.3%), and mixed/other race,
ethnicity, or origin (6.3%). The smallest
representation among tenure-track faculty
are teachers of Black or African and of
Hispanic, Latino, or Spanish origins, who
account for 0.6% and 0.8%, respectively.

Asianorigin2.7%
2.7%
Middle Easternor Eastern or
MiddleNorth African
North African

6.3%
Mixed/other6.3%
race,
ethnicity, or Mixed/otherorigin
race,
67.0%
White

8.3%
Other races

67.0%
White

ethnicity, or origin

8.3%
Other races
Figure 6.2.3

B L AC K I N A I
Black in AI (BAI), founded in 2017 by Timnit Gebru
and Rediet Abebe, is a multi-institutional andtranscontinental initiative that aims to increase thepresence of Black people in the field of AI. As of 2020, BAI
has around 3,000 community members and allies, has
held more than 10 workshops at major AI conferences,and has helped increase the number of Black people

participating at major AI conferences globally 40-fold.
Figure 6.2.4 shows the number of attendees, submitted
papers, and accepted papers from the annual Blackin AI Workshop, which is co-located with NeurIPS.4
The numbers of attendees in 2019 are 2.6 times higher
than in 2017, while the number of accepted papers
is 2.1 times higher.

NUMBER OF ATTENDEES, SUBMITTED PAPERS, and ACCEPTED PAPERS at BLACK in AI WORKSHOP CO-LOCATED
with NEURIPS, 2017-19
Source: Black in AI, 2020 | Chart: 2021 AI Index Report

Number of
Attendees

Submitted
Papers2017
2018

Accepted
Papers

2019
0

100

200

300
Number of Attendees and Papers

400

500
Figure 6.2.4

3 The survey was distributed to 73 universities online over three waves from November 2020 to January 2021 and completed by 18 universities, a 24.7% response rate. The 18 universities are Belgium:
Katholieke Universiteit Leuven; Canada: McGill University; China: Shanghai Jiao Tong University, Tsinghua University; Germany: Ludwig Maximilian University of Munich, Technical University of Munich;Russia: Higher School of Economics, Moscow Institute of Physics and Technology; Switzerland: École Polytechnique Fédérale de Lausanne; United Kingdom: University of Cambridge; United States:
California Institute of Technology, Carnegie Mellon University (Department of Machine Learning), Columbia University, Harvard University, Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University.
4 The 2020 data are clearly affected by the pandemic and not included as a result. For more information, see the Black in AI impact report.

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

146

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6. 3 G E N D E R I D E N T I T Y
A N D S E X UA L
O R I E N TAT I O N I N A I

6.3 GENDER IDENTITY ANDSEXUAL ORIENTATION IN AI
QUEER IN AI
This section presents data from a membership survey
by Queer in AI (QAI), 5 an organization that aims to makethe AI/ML community one that welcomes, supports,
and values queer scientists. Founded in 2018 by William
Agnew, Raphael Gontijo Lopes, and Eva Breznik, QAI
builds a visible community of queer and ally AI/ML
scientists through meetups, poster sessions, mentoring,and other initiatives.

Demographics Breakdown
According to the 2020 survey, with around 100 responses,

about 31.5% of respondents identify as gay, followedby bisexual, queer, and lesbian (Figure 6.3.1); around
37.0% and 26.1% of respondents identify as cis maleand cis female, respectively, followed by gender queer,
gender fluid, nonbinary, and others (Figure 6.3.2). Transfemale and male account for 5.0% and 2.5% of total
members, respectively. Moreover, the past three years ofsurveys show that students make up the majority of QAI
members—around 41.7% of all respondents on average(Figure 6.3.3), followed by junior-level professionals inacademia or industry.

QAI MEMBERSHIP SURVEY: WHAT IS YOUR SEXUAL ORIENTATION, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report

Gay
Bisexual
Queer
LesbianStraight
Asexual
Pansexual
Others
0%

5%

10%

15%

20%

25%

30%

% of Respondents
Figure 6.3.1

5 Queer in AI presents the survey results at its workshop at the annual NeurIPS conference.

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

147

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

6. 3 G E N D E R I D E N T I T Y
A N D S E X UA L
O R I E N TAT I O N I N A I

QAI MEMBERSHIP SURVEY: WHAT IS YOUR GENDER IDENTITY, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report

Cis Male

Cis Female

Gender Queer

Gender Fluid

NonBinary and Others

Trans Female
0%

5%

10%

15%

20%

25%

30%

35%

% of Respondents
Figure 6.3.2

QAI MEMBERSHIP SURVEY: HOW WOULD YOU DESCRIBE YOUR POSITION, 2018-20
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report

Student

JuniorAcademic

JuniorIndustry

2018
2019
2020

Others

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

% of Respondents
Figure 6.3.3

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

148

Artificial IntelligenceIndex Report 2021

CHAPTER 6:
DIVERSITY IN AI

Experience as Queer Practitioners

6. 3 G E N D E R I D E N T I T Y
A N D S E X UA L
O R I E N TAT I O N I N A I

Among surveyed QAI
members, 81.4% regard
the lack of role models
as being a major obstacle
for their careers, and70.9% think the lack ofcommunity contributes tothe same phenomenon.

QAI also surveyed its members on their experiences
as queer AI/ML practitioners. As shown in Figure
6.3.4, 81.4% regard the lack of role models as beinga major obstacle for their careers, and 70.9% thinkthe lack of community contributes to the same
phenomenon. Almost half the respondents also
view the lack of inclusiveness in the field as an
obstacle. Moreover, more than 40% of QAI members
have experienced discrimination or harassment
as a queer person at work or school (Figure 6.3.5).
Around 9.7% have encountered discrimination orharassment on more than five occasions.

QAI MEMBERSHIP SURVEY: WHAT ARE OBSTACLES YOU HAVE FACED in BECOMING a QUEER AI/ML
PRACTITIONER, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report

Lack of Role Models

Lack of Community

Lack of InclusivenessLack of Work/School Support

Economic Hardship Now

Harrassment/Discrimination
0%

10%

20%

30%

40%

50%

60%

70%

80%

% of Respondents
Figure 6.3.4

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

149

Artificial IntelligenceIndex Report 2021

6. 3 G E N D E R I D E N T I T Y
A N D S E X UA L
O R I E N TAT I O N I N A I

CHAPTER 6:
DIVERSITY IN AI

More than 40% of QAI members have experienced
discrimination or harassment as a queer person at workor school. Around 9.7% have encountered discriminationor harassment on more than five occasions.

QAI MEMBERSHIP SURVEY: HAVE YOU EXPERIENCED DISCRIMINATION/HARASSMENT as a QUEER PERSON at YOURJOB or SCHOOL, 2020
Source: Queer in AI, 2020 | Chart: 2021 AI Index Report

0 times

1 time2 times5+ times

Others

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

% of Respondents
Figure 6.3.5

CHAPTER 6 PREVIEWTA B L E O F C O N T E N T S

150

Artificial IntelligenceIndex Report 2021

CHAPTER 7:

AI Policy andNational Strategies

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

151

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

CHAPTER 7:

Chapter Preview
Overview

153

Working Group

165

Chapter Highlights

154

Summits and Meetings

166

Bilateral Agreements

166
167

7.1 NATIONAL AND REGIONALAI STRATEGIES

155

7.3 U.S. PUBLIC INVESTMENT IN AI

Published Strategies

156

Federal Budget for Non-Defense AI R&D 167

2017

156

2018

157

U.S. Department of Defense
Budget Request

168

2019

159

2020

161

U.S. Government AI-Related
Contract Spending

169

Total Contract Spending

169

Contract Spending by
Department and Agency

169

7.4 AI AND POLICYMAKING

171

Legislation Records on AI

171

Strategies in Development(as of December 2020)
Strategies in Public Consultation
Strategies Announced
Highlight: National AI Strategies
and Human Rights

162
162
163
164

7.2 INTERNATIONALCOLLABORATION ON AI

165

Intergovernmental Initiatives

165

U.S. Congressional Record

172

Mentions of AI and ML in
Congressional/Parliamentary
Proceedings

172

Central Banks

174

U.S. AI Policy Papers

176

ACCESS THE PUBLIC DATACHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

152

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E SOV E R V I E W

Overview
AI is set to shape global competitiveness over the coming decades, promising
to grant early adopters a significant economic and strategic advantage. To
date, national governments and regional and intergovernmental organizations
have raced to put in place AI-targeted policies to maximize the promise of thetechnology while also addressing its social and ethical implications.
This chapter navigates the landscape of AI policymaking and tracks efforts taking
place on the local, national, and international levels to help promote and govern AI
technologies. It begins with an overview of national and regional AI strategies andthen reviews activities on the intergovernmental level. The chapter then takes acloser look at public investment in AI in the United States as well as how legislative
bodies, central banks, and nongovernmental organizations are responding to thegrowing need to institute a policy framework for AI technologies.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

153

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• Since Canada published the world’s first national AI strategy in 2017, more than 30 othercountries and regions have published similar documents as of December 2020.
• The launch of the Global Partnership on AI (GPAI) and Organisation for Economic
Co-operation and Development (OECD) AI Policy Observatory and Network of Experts
on AI in 2020 promoted intergovernmental efforts to work together to support thedevelopment of AI for all.
• In the United States, the 116th Congress was the most AI-focused congressional session inhistory. The number of mentions of AI by this Congress in legislation, committee reports, andCongressional Research Service (CRS) reports is more than triple that of the 115th Congress.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

154

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E SThis section presents an overview of select national and regional AI strategies from around the world, including details on
the strategies for G20 countries, Estonia, and Singapore as well as links to strategy documents for many others. Sources
include websites of national or regional governments, the OECD AI Policy Observatory (OECD.AI), and news coverage. “AIstrategy” is defined as a policy document that communicates the objective of supporting the development of AI while also
maximizing the benefits of AI for society. Excluded are broader innovation or digital strategy documents which do not focus
predominantly on AI, such as Brazil’s E-Digital Strategy and Japan’s Integrated Innovation Strategy.

7.1 NATIONAL AND REGIONALAI STRATEGIESTo guide and foster the development of AI, countries and regions around the world are establishing strategies andinitiatives to coordinate governmental and intergovernmental efforts. Since Canada published the world’s first national
AI strategy in 2017, more than 30 other countries and regions have published similar documents as of December 2020.

COUNTRIESWITH PUBLISHEDAI STRATEGIES: 32

COUNTRIESDEVELOPING
AI STRATEGIES: 22

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

155

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

Published Strategies
2017

Canada• AI Strategy: Pan Canadian AI Strategy• Responsible Organization: Canadian Institute forAdvanced Research (CIFAR)• Highlights: The Canadian strategy emphasizes
developing Canada’s future AI workforce, supporting major
AI innovation hubs and scientific research, and positioning
the country as a thought leader in the economic, ethical,
policy, and legal implications of artificial intelligence.
• Funding (December 2020 conversion rate): CAD 125
million (USD 97 million)• In November 2020, CIFAR published its most recent
annual report, titled “AICAN,” which tracks progress on
implementing its national strategy, which highlighted
substantial growth in Canada’s AI ecosystem, as well
as research and activities related to healthcare and AI’s
impact on society, among other outcomes of the strategy.

China• AI Strategy: A Next Generation Artificial Intelligence
Development Plan
• Responsible Organization: State Council for the People’s
Republic of China• Highlights: China’s AI strategy is one of the mostcomprehensive in the world. It encompasses areas
including R&D and talent development througheducation and skills acquisition, as well as ethical norms
and implications for national security. It sets specifictargets, including bringing the AI industry in line withcompetitors by 2020; becoming the global leader in fields
such as unmanned aerial vehicles (UAVs), voice andimage recognition, and others by 2025; and emerging as
the primary center for AI innovation by 2030.
• Funding: N/A• Recent Updates: China established a New Generation
AI Innovation and Development Zone in February 2019and released the “Beijing AI Principles” in May 2019 withCHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

a multi-stakeholder coalition consisting of academic
institutions and private-sector players such as Tencent
and Baidu.

Japan• AI Strategy: Artificial Intelligence Technology Strategy• Responsible Organization: Strategic Council for AI
Technology• Highlights: The strategy lays out three discrete phases ofAI development. The first phase focuses on the utilizationof data and AI in related service industries, the secondon the public use of AI and the expansion of service
industries, and the third on creating an overarching
ecosystem where the various domains are merged.
• Funding: N/A• Recent Updates: In 2019, the Integrated Innovation
Strategy Promotion Council launched another AI strategy,
aimed at taking the next step forward in overcoming
issues faced by Japan and making use of the country’s
strengths to open up future opportunities.

OthersFinland: Finland’s Age of Artificial Intelligence
United Arab Emirates: UAE Strategy for Artificial
Intelligence156

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

Published Strategies
2018

European Union• AI Strategy: Coordinated Plan on Artificial Intelligence• Responsible Organization: European Commission• Highlights: This strategy document outlines thecommitments and actions agreed on by EU member
states, Norway, and Switzerland to increase investment
and build their AI talent pipeline. It emphasizes the valueof public-private partnerships, creating European data
spaces, and developing ethics principles.
• Funding (December 2020 conversation rate): At least
EUR 1 billion (USD 1.1 billion) per year for AI research andat least EUR 4.9 billion (USD 5.4 billion) for other aspectsof the strategy• Recent updates: A first draft of the ethics guidelines was
released in June 2018, followed by an updated version in
April 2019.

France• AI Strategy: AI for Humanity: French Strategy for Artificial
Intelligence• Responsible Organizations: Ministry for Higher
Education, Research and Innovation; Ministry of Economy
and Finance; Directorate General for Enterprises; Public
Health Ministry; Ministry of the Armed Forces; National
Research Institute for Digital Sciences; Interministerial
Director of the Digital Technology and the Information
and Communication System• Highlights: The main themes include developing
an aggressive data policy for big data; targeting four
strategic sectors, namely health care, environment,
transport, and defense; boosting French efforts inresearch and development; planning for the impact of AI
on the workforce; and ensuring inclusivity and diversity
within the field.
• Funding (December 2020 conversion rate): EUR 1.5
billion (USD 1.8 billion) up to 2022

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S• Recent Updates: The French National Research Institutefor Digital Sciences (Inria) has committed to playing acentral role in coordinating the national AI strategy andwill report annually on its progress.

Germany• AI Strategy: AI Made in Germany• Responsible Organizations: Federal Ministry of
Education and Research; Federal Ministry for Economic
Affairs and Energy; Federal Ministry of Labour and Social
Affairs• Highlights: The focus of the strategy is on cementing
Germany as a research powerhouse and strengthening
the value of its industries. There is also an emphasison the public interest and working to better the lives ofpeople and the environment.
• Funding (December 2020 conversion rate): EUR 500
million (USD 608 million) in the 2019 budget and EUR
3 billion (USD 3.6 billion) for the implementation up to2025• Recent Updates: In November 2019, the government
published an interim progress report on the Germany AI
strategy.

157

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

2018 (continued)

India• AI Strategy: National Strategy on
Artificial Intelligence: #AIforAll• Responsible Organization: National Institution forTransforming India (NITI Ayog)• Highlights: The Indian strategy focuses on both
economic growth and ways to leverage AI to increasesocial inclusion, while also promoting research toaddress important issues such as ethics, bias, andprivacy related to AI. The strategy emphasizes sectors
such as agriculture, health, and education, where public
investment and government initiative are necessary.
• Funding (December 2020 conversion rate): INR 7000
crore (USD 949 million)• Recent Updates: In 2019, the Ministry of Electronics andInformation Technology released its own proposal toset up a national AI program with an allocated INR 400
crore (USD 54 million). The Indian government formed
a committee in late 2019 to push for an organized AI
policy and establish the precise functions of government
agencies to further India’s AI mission.

Mexico• AI Strategy: Artificial Intelligence Agenda MX(2019 agenda-in-brief version)• Responsible Organization: IA2030Mx, Economía• Highlights: As Latin America’s first strategy, the Mexican
strategy focuses on developing a strong governanceframework, mapping the needs of AI in various industries,
and identifying governmental best practices with anemphasis on developing Mexico’s AI leadership.
• Funding: N/A• Recent Updates: According to the Inter-American
Development Bank’s recent fAIr LAC report, Mexico is in
the process of establishing concrete AI policies to further
implementation.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

United Kingdom• AI Strategy: Industrial Strategy: Artificial Intelligence
Sector Deal
• Responsible Organization: Office for Artificial
Intelligence (OAI)• Highlights: The U.K. strategy emphasizes a strong
partnership between business, academia, and thegovernment and identifies five foundations for a
successful industrial strategy: becoming the world’s mostinnovative economy, creating jobs and better earnings
potential, infrastructure upgrades, favorable businessconditions, and building prosperous communities
throughout the country.
• Funding (December 2020 conversion rate): GBP 950
million (USD 1.3 billion)• Recent Updates: Between 2017 and 2019, the U.K.’s
Select Committee on AI released an annual report on the
country’s progress. In November 2020, the government
announced a major increase in defense spending of
GBP 16.5 billion (USD 21.8 billion) over four years, witha major emphasis on AI technologies that promise to
revolutionize warfare.

OthersSweden: National Approach to Artificial Intelligence
Taiwan: Taiwan AI Action Plan

158

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

Published Strategies
2019

Estonia• AI Strategy: National AI Strategy 2019–2021• Responsible Organization: Ministry of Economic Affairs
and Communications (MKM)• Highlights: The strategy emphasizes actions necessary
for both the public and private sectors to take to increase
investment in AI research and development, while also
improving the legal environment for AI in Estonia. In
addition, it hammers out the framework for a steeringcommittee that will oversee the implementation andmonitoring of the strategy.
• Funding (December 2020 conversion rate): EUR 10
million (USD 12 million) up to 2021
• Recent Updates: The Estonian government released an
update on the AI taskforce in May 2019.

Russia• AI Strategy: National Strategy for the Development of
Artificial Intelligence• Responsible Organizations: Ministry of Digital
Development, Communications and Mass Media;
Government of the Russian Federation• Highlights: The Russian AI strategy places a strongemphasis on its national interests and lays down
guidelines for the development of an “informationsociety” between 2017 and 2030. These include anational technology initiative, departmental projects
for federal executive bodies, and programs such as the
Digital Economy of the Russian Federation, designed to
implement the AI framework across sectors.
• Funding: N/A• Recent Updates: In December 2020, Russian president
Vladmir Putin took part in the Artificial Intelligence
Journey Conference, where he presented four ideas for AI
policies: establishing experimental legal frameworks for

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

the use of AI, developing practical measures to introduce
AI algorithms, providing neural network developers withcompetitive access to big data, and boosting private
investment in domestic AI industries.

Singapore• AI Strategy: National Artificial Intelligence Strategy• Responsible Organization: Smart Nation and Digital
Government Office (SNDGO)• Highlights: Launched by Smart Nation Singapore, agovernment agency that seeks to transform Singapore’s
economy and usher in a new digital age, the strategy
identifies five national AI projects in the following fields:
transport and logistics, smart cities and estates, healthcare, education, and safety and security.
• Funding (December 2020 conversion rate): While the2019 strategy does not mention funding, in 2017 thegovernment launched its national program, AI Singapore,with a pledge to invest SGD 150 million (USD 113 million)
over five years.
• Recent Updates: In November 2020, SNDGO published
its inaugural annual update on the Singaporean
government’s data protection efforts. It describes themeasures taken to date to strengthen public sector data
security and to safeguard citizens’ private data.

159

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

2019 (continued)

United States• AI Strategy: American AI Initiative• Responsible Organization: The White House• Highlights: The American AI Initiative prioritizes
the need for the federal government to invest in AI
R&D, reduce barriers to federal resources, and ensuretechnical standards for the safe development, testing,and deployment of AI technologies. The White Housealso emphasizes developing an AI-ready workforce andsignals a commitment to collaborating with foreign
partners while promoting U.S. leadership in AI. The
initiative, however, lacks specifics on the program’stimeline, whether additional research will be dedicatedto AI development, and other practical considerations.
• Funding: N/A• Recent Updates: The U.S. government released its
year one annual report in February 2020, followed in
November by the first guidance memorandum for federal
agencies on regulating artificial intelligence applicationsin the private sector, including principles that encourage
AI innovation and growth and increase public trust andconfidence in AI technologies. The National Defense
Authorization Act (NDAA) for Fiscal Year 2021 called for a
National AI Initiative to coordinate AI research and policy
across the federal government.

South Korea• AI Strategy: National Strategy for Artificial Intelligence• Responsible Organization: Ministry of Science, ICT and
Future Planning (MSIP)• Highlights: The Korean strategy calls for plans tofacilitate the use of AI by businesses and to streamline
regulations to create a more favorable environment forthe development and use of AI and other new industries.
The Korean government also plans to leverage its
dominance in the global supply of memory chips to build
the next generation of smart chips by 2030.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S• Funding (December 2020 conversion rate):
KRW 2.2 trillion (USD 2 billion)• Recent Updates: N/A

OthersColombia: National Policy for Digital Transformation
and Artificial Intelligence
Czech Republic: National Artificial Intelligence
Strategy of the Czech Republic
Lithuania: Lithuanian Artificial Intelligence Strategy: A
Vision for the Future
Luxembourg: Artificial Intelligence: A Strategic Vision
for Luxembourg
Malta: Malta: The Ultimate AI Launchpad
Netherlands: Strategic Action Plan for Artificial
Intelligence
Portugal: AI Portugal 2030
Qatar: National Artificial Intelligence for Qatar

160

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

Published Strategies
2020

Indonesia• AI Strategy: National Strategy for the Development of
Artificial Intelligence (Stranas KA)• Responsible Organizations: Ministry of Research
and Technology (Menristek), National Research andInnovation Agency (BRIN), Agency for the Assessment andApplication of Technology (BPPT)• Strategy Highlights: The Indonesian strategy aims
to guide the country in developing AI between 2020
and 2045. It focuses on education and research, healthservices, food security, mobility, smart cities, and public
sector reform.
• Funding: N/A• Recent Updates: None

Saudi Arabia• AI Strategy: National Strategy on Data and AI (NSDAI)• Responsible Organization: Saudi Data and Artificial
Intelligence Authority (SDAIA)• Highlights: As part of an effort to diversify the country’s
economy away from oil and boost the private sector, the
NSDAI aims to accelerate AI development in five criticalsectors: health care, mobility, education, government,
and energy. By 2030, Saudi Arabia intends to train 20,000
data and AI specialists, attract USD 20 billion in foreign
and local investment, and create an environment thatwill attract at least 300 AI and data startups.
• Funding: N/A• Recent Updates: During the summit where theSaudi government released its strategy, the country’s
National Center for Artificial Intelligence (NCAI) signed
collaboration agreements with China’s Huawei andAlibaba Cloud to design AI-related Arabic-language
systems.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

OthersHungary: Hungary’s Artificial Intelligence Strategy
Norway: National Strategy for Artificial Intelligence
Serbia: Strategy for the Development of Artificial
Intelligence in the Republic of Serbia for the Period
2020–2025
Spain: National Artificial Intelligence Strategy

161

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E SStrategies in Development(AS OF DECEMBER 2020)

Strategies in Public Consultation
Brazil• AI Strategy Draft: Brazilian Artificial Intelligence Strategy• Responsible Organization: Ministry of Science,
Technology and Innovation (MCTI)• Highlights: Brazil’s national AI strategy was announced
in 2019 and is currently in the public consultation stage.
According to the OECD, the strategy aims to cover
relevant topics bearing on AI, including its impact on theeconomy, ethics, development, education, and jobs, andto coordinate specific public policies addressing such
issues.
• Funding: N/A• Recent Updates: In October 2020, the country’s largest
research facility dedicated to AI was launched incollaboration with IBM, the University of São Paulo, andthe São Paulo Research Foundation.

OthersCyprus: National Strategy for Artificial Intelligence
Ireland: National Irish Strategy on Artificial Intelligence
Poland: Artificial Intelligence Development Policy in
PolandUruguay: Artificial Intelligence Strategy for Digital

Government

Italy• AI Strategy Draft: Proposal for an Italian Strategy for
Artificial Intelligence• Responsible Organization: Ministry of Economic
Development (MISE)• Highlights: This document provides the proposed
strategy for the sustainable development of AI, aimed
at improving Italy’s competitiveness in AI. It focuses on
improving AI-based skills and competencies, fostering AI
research, establishing a regulatory and ethical framework
to ensure a sustainable ecosystem for AI, and developinga robust data infrastructure to fuel these developments.
• Funding (December 2020 conversion rate): EUR 1
billion (USD 1.1 billion) through 2025 and expected
matching funds from the private sector, bringing the total
investment to EUR 2 billion.
• Recent Updates: None

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

162

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E SStrategies Announced
Argentina• Related Document: N/A• Responsible Organization: Ministry of Science,
Technology and Productive Innovation (MINCYT)• Status: Argentina’s AI plan is a part of the Argentine
Digital Agenda 2030 but has not yet been published. It is
intended to cover the decade between 2020 and 2030,
and reports indicate that it has the potential to reap hugebenefits for the agricultural sector.

Australia• Related Documents: Artificial Intelligence Roadmap /An AI Action Plan for all Australians• Responsible Organizations: Commonwealth Scientificand Industrial Research Organisation (CSIRO), Data 61,
and the Australian government• Status: The Australian government published a road
map in 2019 (in collaboration with the national scienceagency, CSIRO) and a discussion paper of an AI action
plan in 2020 as frameworks to develop a national
AI strategy. In its 2018–19 budget, the Australian
government earmarked AUD 29.9 million (USD 22.2
million [December 2020 conversation rate]) over four
years to strengthen the country’s capabilities in AI andmachine learning (ML). In addition, CSIRO published aresearch paper on Australia’s AI Ethics Framework in 2019and launched a public consultation, which is expected to
produce a forthcoming strategy document.

Turkey• Related Document: N/A• Responsible Organizations: Presidency of the Republic
of Turkey Digital Transformation Office; Ministry ofIndustry and Technology; Scientific and TechnologicalResearch Council of Turkey; Science, Technology andInnovation Policies Council• Status: The strategy has been announced but not yet
published. According to media sources, it will focus

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

on talent development, scientific research, ethics andinclusion, and digital infrastructure.

OthersAustria: Artificial Intelligence Mission Austria(official report)
Bulgaria: Concept for the Development of Artificial
Intelligence in Bulgaria Until 2030 (concept document)
Chile: National AI Policy (official announcement)
Israel: National AI Plan (news article)
Kenya: Blockchain and Artificial Intelligence Taskforce(news article)
Latvia: On the Development of Artificial Intelligence
Solutions (official report)
Malaysia: National Artificial Intelligence (Al) Framework(news article)
New Zealand: Artificial Intelligence: Shaping a Future
New Zealand (official report)
Sri Lanka: Framework for Artificial Intelligence (newsarticle)Switzerland: Artificial Intelligence (official guidelines)
Tunisia: National Artificial Intelligence Strategy (taskforce announced)
Ukraine: Concept of Artificial Intelligence Development
in Ukraine AI (concept document)
Vietnam: Artificial Intelligence Development Strategy(official announcement)163

Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7.1 N AT I O N A L
AND REGIONAL
A I S T R AT E G I E S

Read more on AI national strategies:• Tim Dutton: An Overview of National AI Strategies• Organisation for Economic Co-operation and Development: OECD AI Policy Observatory• Canadian Institute for Advanced Research: Building an AI World, Second Edition• Inter-American Development Bank: Artificial Intelligence for Social Good in Latin America and the Caribbean:
The Regional Landscape and 12 Country Snapshots

National AI Strategies and Human Rights
In 2020, Global Partners Digital and Stanford’s
Global Digital Policy Incubator published areport examining governments’ national AI
strategies from a human rights perspective,
titled “National Artificial Intelligence Strategies
and Human Rights: A Review.” The reportassesses the extent to which governments
and regional organizations have incorporated
human rights considerations into their national
AI strategies and made recommendations topolicymakers looking to develop or review AI
strategies in the future.
The report found that among the 30 states andtwo regional strategies (from the European
Union and the Nordic-Baltic states), a numberof strategies refer to the impact of AI on humanrights, with the right to privacy as the mostcommonly mentioned, followed by equality
and nondiscrimination (Table 6.1.1). However,very few strategy documents provide deep
analysis or concrete assessment of the impact
of AI applications on human rights. Specifics
as to how and the depth to which humanrights should be protected in the context of
AI is largely missing, in contrast to the level of
specificity on other issues such as economic
competitiveness and innovation advantage.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

Table 7.1.1: Mapping human rights
referenced in national AI strategies
HUMAN RIGHTS
MENTIONEDSTATES/REGIONALORGANIZATIONS

The right to privacy

Australia, Belgium, China,
Czech Republic, Germany,
India, Italy, Luxembourg, Malta,
Netherlands, Norway, Portugal,
Qatar, South Korea, UnitedStates

The right
to equality/
nondiscrimination

Australia, Belgium, Czech
Republic, Denmark, Estonia, EU,
France, Germany, Italy, Malta,
Netherlands, Norway

The right to aneffective remedy

Australia (responsibility
and ability to hold humans
responsible), Denmark, Malta,
Netherlands

The rights tofreedom of thought,
expression,and access to
information

France, Netherlands,
RussiaThe right to work

France, Russia

164Artificial IntelligenceIndex Report 2021

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7. 2 I N T E R N AT I O N A L
C O L L A B O R AT I O N
ON AI

7.2 INTERNATIONALCOLLABORATION ON AI
Given the scale of the opportunities and the challenges
presented by AI, a number of international efforts haverecently been announced that aim to develop multilateral
AI strategies. This section provides an overview of thoseinternational initiatives from governments committed toworking together to support the development of AI for all.
These multilateral initiatives on AI suggest thatorganizations are taking a variety of approaches totackle the practical applications of AI and scale thosesolutions for maximum global impact. Many countries
turn to international organizations for global AI norm
formulation, while others engage in partnerships orbilateral agreements. Among the topics under discussion,
the ethics of AI—or the ethical challenges raised by current
and future applications of AI—stands out as a particularfocus area for intergovernmental efforts.
Countries such as Japan, South Korea, the United
Kingdom, the United States, and members of the European
Union are active participants of intergovernmental
efforts on AI. A major AI powerhouse, China, on the other
hand, has opted to engage in a number of science andtechnology bilateral agreements that stress cooperation
on AI as part of the Digital Silk Road under the Beltand Road (BRI) initiative framework. For example, AI is
mentioned in China’s economic cooperation under the BRI
Initiative with the United Arab Emirates.

I N T E R G OV E R N M E N TA LI N I T I AT I V E SIntergovernmental working groups consist of experts andpolicymakers from member states who study and reporton the most urgent challenges related to developing anddeploying AI and then make recommendations based on
their findings. These groups are instrumental in identifying
and developing strategies for the most pressing issues in AI
technologies and their applications.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

Working Groups
Global Partnership on AI (GPAI)• Participants: Australia, Brazil, Canada, France, Germany,
India, Italy, Japan, Mexico, the Netherlands, New
Zealand, South Korea, Poland, Singapore, Slovenia,
Spain, the United Kingdom, the United States, and theEuropean Union (as of December 2020)• Host of Secretariat: OECD
• Focus Areas: Responsible AI; data governance; the futureof work; innovation and commercialization• Recent Activities: Two International Centres ofExpertise—the International Centre of Expertise inMontreal for the Advancement of Artificial Intelligenceand the French National Institute for Research in Digital
Science and Technology (INRIA) in Paris—are supporting
the work in the four focus areas and held the Montreal
Summit 2020 in December 2020. Moreover, the data
governance working group published the beta version of
the group’s framework in November 2020.
OECD Network of Experts on AI (ONE AI)• Participants: OECD countries• Host: OECD
• Focus Areas: Classification of AI; implementing
trustworthy AI; policies for AI; AI compute• Recent Activities: ONE AI convened its first meeting in
February 2020, when it also launched the OECD AI Policy
Observatory. In November 2020, the working group on
the classification of AI presented the first look at an AI
classification framework based on OECD’s definition of AI
divided into four dimensions (context, data and input, AI
model, task and output) that aims to guide policymakers
in designing adequate policies for each type of AI system.
High-Level Expert Group on Artificial Intelligence (HLEG)• Participants: EU countries• Host: European Commission• Focus Areas: Ethics guidelines for trustworthy AI•R ecent Activities: Since its launch at the recommendation

165

Artificial IntelligenceIndex Report 2021of the EU AI strategy in 2018, HLEG presented the EU Ethics
Guidelines for Trustworthy Artificial Intelligence and aseries of policy and investment recommendations, aswell as an assessment checklist related to the guidelines.
Ad Hoc Expert Group (AHEG) for the Recommendationon the Ethics of Artificial Intelligence• Participants: United Nations Educational, Scientific andCultural Organization (UNESCO) member states• Host: UNESCO•F
 ocus Areas: Ethical issues raised by the development
and use of AI• Recent Activities: The AHEG produced a revised first draft
Recommendation on the Ethics of Artificial Intelligence,
which was transmitted in September 2020 to Member Statesof UNESCO for their comments by December 31, 2020.

Summits and Meetings
AI for Good Global Summit• Participants: Global (with the United Nations and its
agencies)• Hosts: International Telecommunication Union, XPRIZE
Foundation• Focus Areas: Trusted, safe, and inclusive development ofAI technologies and equitable access to their benefits
AI Partnership for Defense• Participants: Australia, Canada, Denmark, Estonia,
Finland, France, Israel, Japan, Norway, South Korea,
Sweden, the United Kingdom, and the United States• Hosts: Joint Artificial Intelligence Center, U.S.
Department of Defense• Focus Areas: AI ethical principles for defense
China-Association of Southeast Asian Nations (ASEAN)
AI Summit• Participants: Brunei, Cambodia, China, Indonesia, Laos,
Malaysia, Myanmar, the Philippines, Singapore, Thailand,
and Vietnam• Hosts: China Association for Science and Technology,
Guangxi Zhuang Autonomous Region, China• Focus Areas: Infrastructure construction, digital
economy, and innovation-driven development

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

7. 2 I N T E R N AT I O N A L
C O L L A B O R AT I O N
ON AI

B I L AT E R A L AG R E E M E N T S
Bilateral agreements focusing on AI are another formof international collaboration that has been gaining in
popularity in recent years. AI is usually included in the
broader context of collaborating on the development ofdigital economies, though India stands apart for investingin developing multiple bilateral agreements specifically
geared toward AI.
India and United Arab Emirates
Invest India and the UAE Ministry of Artificial Intelligencesigned a memorandum of understanding in July 2018
to collaborate on fostering innovative AI ecosystems
and other policy concerns related to AI. Two countries
will convene a working committee aimed at increasing
investment in AI startups and research activities inpartnership with the private sector.
India and Germany
It was reported in October 2019 that India and Germany
likely will sign an agreement including partnerships on theuse of artificial intelligence (especially in farming).
United States and United Kingdom
The U.S. and the U.K. announced a declaration in
September 2020, through the Special RelationshipEconomic Working Group, that the two countries willenter into a bilateral dialogue on advancing AI in line withshared democratic values and further cooperation in AI
R&D efforts.
India and Japan
India and Japan were said to have finalized an agreementin October 2020 that focuses on collaborating on digitaltechnologies, including 5G and AI.
French and Germany
France and Germany signed a road map for a FrancoGerman Research and Innovation Network on artificial
intelligence as part of the Declaration of Toulousein October 2019 to advance European efforts in thedevelopment and application of AI, taking into accountethical guidelines.

166

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7. 3 U.S. P U B L I C
INVESTMENT
IN AI

This section examines public investment in AI in the United States based on data from the U.S. Networking and Information
Technology Research and Development (NITRD) program and Bloomberg Government.

7.3 U.S. PUBLIC INVESTMENT IN AI
In September 2019, the White House
National Science and Technology Council
released a report attempting to total
up all public-sector AI R&D funding, thefirst time such a figure was published.
This funding is to be disbursed as grants
for government laboratories or research
universities or in the form of government
contracts. These federal budget figures,
however, do not include substantial AI
R&D investments by the Department of
Defense (DOD) and the intelligence sector,
as they were withheld from publication for
national security reasons.
As shown in Figure 7.3.1, federal civilian
agencies—those agencies that are not partof the DOD or the intelligence sector—
allocated USD 973.5 million to AI R&D
for FY 2020, a figure that rose to USD 1.1
billion once congressional appropriations
and transfers were factored in. For FY
2021, federal civilian agencies budgeted
USD 1.5 billion, which is almost 55%
higher than its 2020 request.

Federal civilian agencies—thoseagencies that are not part of the
DOD or the intelligence sector—
allocated USD 973.5 million to
AI R&D for FY 2020, a figurethat rose to USD 1.1 billion once
congressional appropriations
and transfers were factored in.
U.S. FEDERAL BUDGET for NON-DEFENSE AI R&D, FY 2020-21
Source: U.S. NITRD Program, 2020 | Chart: 2021 AI Index Report

1,500

Budget (in Millions of U.S. Dollars)

FEDERAL BUDGET FOR
N O N - D E F E N S E A I R& D

1,000

500

0
2020 (Request)

2020 (Enacted)

2021 (Request)
Figure 7.3.1

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

167

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

U.S. D E PA R T M E N T O F D E F E N S E A I
R& D B U D G E T R E Q U E S T

7. 3 U.S. P U B L I C
INVESTMENT
IN AI

The top five projects set to receive the highest amount of
AI R&D investment in FY 2021:• Rapid Capability Development and Maturation, by the
U.S. Army (USD 284.2 million)• Counter WMD Technologies and Capabilities
Development, by the DOD Threat Reduction Agency(USD 265.2 million)• Algorithmic Warfare Cross-Functional Team (Project
Maven), by the Office of the Secretary of Defense (USD
250.1 million)• Joint Artificial Intelligence Center (JAIC), by the Defense
Information Systems Agency (USD 132.1 million)• High Performance Computing Modernization Program,by the U.S. Army (USD 99.6 million)
In addition, the Defense Advanced Research Projects
Agency (DARPA) alone is investing USD 568.4 million in AI
R&D, an increase of USD 82 million from FY 2020.

While the official DOD budget is not publicly available,
Bloomberg Government has analyzed the department’s
publicly available budget request for research,development, test, and evaluation (RDT&E)— data thatsheds light on its spending on AI R&D.
With 305 unclassified DOD R&D programs specifying the useof AI or ML technologies, the combined U.S. military budget
for AI R&D in FY 2021 is USD 5.0 billion (Figure 7.3.2). This
figure appears consistent with the USD 5.0 billion enacted
the previous year. However, the FY 2021 figure reflects
a budget request, rather than a final enacted budget.
As noted above, once congressional appropriations arefactored in, the true level of funding available to DOD AI R&D
programs in FY 2021 may rise substantially.

U.S. DOD BUDGET for AI-SPECIFIC RESEARCH DEVELOPMENT, TEST, and EVALUATION (RDT&E), FY 2018-20
Sources: Bloomberg Government & U.S. Department of Defense, 2020 | Chart: 2021 AI Index Report

Budget (in Millions of U.S. Dollars)

5,000

4,000

3,000

DOD Reported
Budget on AI R&D

2,000

DOD Reported
Budget on AI R&D

927

1,000

841

0
2018 (Enacted)

2019 (Enacted)

2020 (Enacted)

2021 (Request)
Figure 7.3.2

Important data caveat: This chart illustrates the challenge of working with contemporary government data sources
to understand spending on AI. By one measure—the requests that include AI-relevant keywords—the DOD is requesting
more than USD 5 billion for AI-specific research development in 2021 . However, DOD’s own accounting produces aradically smaller number: USD 841 million. This relates to the issue of defining where an AI system ends and anothersystem begins; for instance, an initiative that uses AI for drones may also count hardware-related expenditures for thedrones within its “AI” budget request, though the AI software component will be much smaller.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

168

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

U.S. G OV E R N M E N T A I - R E L AT E D
C O N T R ACT S P E N D I N G
Another indicator of public investment in AI technologies isthe level of spending on government contracts across thefederal government. Contracting for products and services
supplied by private businesses typically occupies the largest
share of an agency’s budget. Bloomberg Government built
a model that captures contract spending on AI technologies
by adding up all contracting transactions that contain aset of more than 100 AI-specific keywords in their titles or
descriptions. The data reveals that the amount the federal
government spends on contracts for AI products andservices has reached an all-time high and shows no sign ofslowing down. However, note that during the procurement
process, vendors may add a bunch of keywords into theirapplications, so some of these things may have a relatively
small AI component relative to other parts of technology.

7. 3 U.S. P U B L I C
INVESTMENT
IN AI

USD 1.5 billion agencies spent in FY 2019 (Figure 7.3.3).
AI spending in 2020 was more than six times higher thanwhat it was just five years ago—about USD 300 million in
FY 2015. However, to put this in perspective, the federal
government spent USD 682 billion on contracts in FY 2020,
so AI currently represents 0.25% of government spending.

Contract Spending by Department and Agency

Total Contract Spending

Figure 7.3.4 shows that in FY 2020, the DOD spent more on
AI-related contracts than any other federal department
or agency (USD 1.4 billion). In second and third place
are NASA (USD 139.1 million) and the Department ofHomeland Security (USD 112.3 million). DOD, NASA, andthe Department of Health and Human Services top thelist for the most contract spending on AI over the past 10
years combined (Figure 7.3.5). In fact, DOD’s total contract
spending on AI from 2001 to 2020 (USD 3.9 billion) is morethan what was spent by the other 44 departments andagencies combined (USD 2.9 billion) over the same period.

Federal departments and agencies spent a combined
USD 1.8 billion on unclassified AI-related contracts in FY
2020. This represents a more than 25% increase from the

Looking ahead, DOD spending on AI contracts is only
expected to grow as the Pentagon’s Joint Artificial
Intelligence Center (JAIC), established in June 2018, isU.S. GOVERNMENT TOTAL CONTRACT SPENDING on AI, FY 2001-20
Source: Bloomberg Government, 2020 | Chart: 2021 AI Index Report

Contract Spending (in Millions of U.S. Dollars)

2,000

1,837

1,500

1,000

500

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0

Figure 7.3.3

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

169

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

still in the early stages of driving DOD’s AI spending. In
2020, JAIC awarded two massive contracts, one to Booz
Allen Hamilton for the five-year, USD 800 million Joint

7. 3 U.S. P U B L I C
INVESTMENT
IN AI

Warfighter program, and another to Deloitte Consulting fora four-year, USD 106 million enterprise cloud environment
for the JAIC, known as the Joint Common Foundation.

TOP 10 CONTRACT SPENDING on AI by U.S. GOVERNMENT DEPARTMENT and AGENCY, 2020
Source: Bloomberg Government, 2020 | Chart: 2021 AI Index Report

Department of Defense (DOD)
National Aeronautics and Space
Administration (NASA)
Department of Homeland Security(DHS)
Department of Health and Human
Services (HHS)
Department of Commerce (DOC)
Department of the Treasury(TREAS)
Department of Veterans A airs
(VA)
Securities and Exchange
Commission (SEC)
Department of Agriculture (USDA)
Department of Justice (DOJ)
0

200

400

600

800

1,000

1,200

Contract Spending (in Millions of U.S. Dollars)

1,400
Figure 7.3.4

TOP 10 CONTRACT SPENDING on AI by U.S. GOVERNMENT DEPARTMENT and AGENCY, 2001-20 (SUM)
Source: Bloomberg Government, 2020 | Chart: 2021 AI Index Report

Department of Defense (DOD)
National Aeronautics andSpace Administration (NASA)
Department of Health andHuman Services (HHS)
Department of the Treasury(TREAS)
Department of Homeland
Security (DHS)
Department of Veterans A airs
(VA)
Department of Commerce(DOC)
Department of Agriculture(USDA)
General Services
Administration (GSA)
Department of State (DOS)
0

500

1000

1500

2000

2500

3000

Contract Spending (in Millions of U.S. Dollars)

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

3500

4000
Figure 7.3.5

170

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7.4 A I A N D
P O L I CY M A K I N G

7.4 AI AND POLICYMAKINGAs AI gains attention and importance, policies andinitiatives related to the technology are becoming higher
priorities for governments, private companies, technicalorganizations, and civil society. This section examines
how three of these four are setting the agenda for AI
policymaking, including the legislative and monetary
authority of national governments, as well as think tanks,
civil society, and the technology and consultancy industry.

number of these records and how that number has
evolved in the last 10 years.
Bloomberg Government identified all legislation (passed
or introduced), reports published by congressional
committees, and CRS reports that referenced one or more
AI-specific keywords. McKinsey & Company searched forthe terms “artificial intelligence” and “machine learning”
on the websites of the U.S. Congressional Record, the U.K.
Parliament, and the Parliament of Canada. For the UnitedStates, each count indicates that AI or ML was mentioned
during a particular event contained in the Congressional
Record, including the reading of a bill; for the U.K. andCanada, each count indicates that AI or ML was mentioned
in a particular comment or remark during the proceedings.1

L E G I S L AT I O N R E C O R D S O N A I
The number of congressional and parliamentary
records on AI is an indicator of governmental interestin developing AI capabilities—and legislating issues
pertaining to AI. In this section, we use data from
Bloomberg and McKinsey & Company to ascertain the

MENTIONS of AI in U.S. CONGRESSIONAL RECORD by LEGISLATIVE SESSION, 2001-20
Source: Bloomberg Government, 2020 | Chart: 2021 AI Index Report486

500
Congressional Research Service Reports
Committee Reports

Number of Mentions

400

243

Legislation
300

70

200
149
66

100

173
39

0

4

7

22

17

15

10

8

16

44

107Th
108Th
109Th
110Th
111Th
112Th
113Th
114Th
115Th
116th
(2001-2002) (2003-2004) (2005-2006) (2007-2008) (2009-2010) (2011-2012) (2013-2014) (2015-2016) (2017-2018) (2019-2020)
Figure 7.4.1

1 If a speaker or member mentioned artificial intelligence (AI) or machine learning (ML) multiple times within remarks, or multiple speakers mentioned AI or ML within the same event, it appears only
once as a result. Counts for AI and ML are separate, as they were conducted in separate searches. Mentions of the abbreviations “AI” or “ML” are not included.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

171

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7.4 A I A N D
P O L I CY M A K I N G

U.S. Congressional Record
subcommittees commissioned 70 reports on AI, while
the CRS, tasked as a fact-finding body for members ofCongress, published 243 about AI or referencing AI.

The 116th Congress (January 1, 2019–January 3, 2021) is
the most AI-focused congressional session in history. The
number of mentions of AI by this Congress in legislation,committee reports, and CRS reports is more than triplethat of the 115th Congress. Congressional interest in AI
has continued to accelerate in 2020. Figure 7.4.1 showsthat during this congressional session, 173 distinct
pieces of legislation either focused on or contained
language about AI technologies, their development,
use, and rules governing them. During that two-year
period, various House and Senate committees andMentions of AI and ML in Congressional/
Parliamentary Proceedings
As shown in Figures 7.4.2–7.4.5, the number of mentionsof artificial intelligence and machine learning in theproceedings of the U.S. Congress and the U.K. parliamentcontinued to rise in 2020, while there were fewer
mentions in the parliamentary proceedings of Canada.

MENTIONS of AI and ML in the PROCEEDINGS of U.S. CONGRESS, 2011-20
Sources: U.S. Congressional Record website, the McKinsey Global Institute, 2020 | Chart: 2021 AI Index Report

140

129
120

120

28

Number of Mentions

28

Machine Learning

100

92

Artificial Intelligence

25

80

60
92

101

40
27

67

20
9

8

0

0

7

1

1

1

6

2011

2012

2013

2014

2015

2016

23

2017

2018

2019

2020
Figure 7.4.2

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

172

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7.4 A I A N D
P O L I CY M A K I N G

MENTIONS of AI and ML in the PROCEEDINGS of U.K. PARLIAMENT, 2011-20
Sources: Parliament of U.K. website, the McKinsey Global Institute, 2020 | Chart: 2021 AI Index Report

300

283
37

250

Number of Mentions

Machine Learning
200

192

Artificial Intelligence

183

34

150

138
246

100

179

158

138

51

50

0

0

2011

42

7

4

1

5

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 7.4.3

MENTIONS of AI and ML in the PROCEEDINGS of CANADIAN PARLIAMENT, 2011-20
Sources: Canadian Parliament website, the McKinsey Global Institute, 2020 | Chart: 2021 AI Index Report40

38

Machine Learning

3

Artificial Intelligence

34

Number of Mentions

30

21

20

18
35

33

21

10

17

2

0

0

0

0

2011

2012

2013

2014

0

0

2015

2016

2017

2018

2019

2020
Figure 7.4.4

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

173

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7.4 A I A N D
P O L I CY M A K I N G

C E N T R A L BA N KS
Central banks play a key role in conducting currency andmonetary policy in a country or a monetary union. As
with many other institutions, central banks are tasked
with integrating AI into their operations and relying on
big data analytics to assist them with forecasting, risk
management, and financial supervision.

Figure 7.4.5 shows a significant increase in the mentionof AI across 16 central banks over the past 10 years,
with the number reaching a peak of 1,020 in 2019. The
sharp decline in 2020 can be explained by the COVID-19
pandemic as most central bank communications focusedon responses to the economic downturn. Moreover,the Federal Reserve in the United States, Norges Bankin Norway, and the European Central Bank top thelist for the most aggregated number of AI mentions incommunications in the past five years (Figure 7.4.6).

Prattle, a leading provider of automated investment
research solutions, monitors mentions of AI in thecommunications of central banks, including meeting
minutes, monetary policy papers, press releases,
speeches, and other official publications.

MENTIONS of AI in CENTRAL BANK COMMUNICATIONS around THE WORLD, 2011-20
Source: Prattle/LiquidNet, 2020 | Chart: 2021 AI Index Report

1,000

Number of Mentions

800

600

400

200

225

0
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 7.4.5

2 See Science & Technology Review and Scientific American for more details.

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

174

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

7.4 A I A N D
P O L I CY M A K I N G

MENTIONS of AI in CENTRAL BANK COMMUNICATIONS around THE WORLD by BANK, 2016-20 (SUM)
Source: Prattle/LiquidNet, 2020 | Chart: 2021 AI Index Report

Federal Reserve
Norges Bank
European Central Bank
Reserve Bank of India
Bank of England
Bank of Israel
Bank of Japan
Bank of Korea
Reserve Bank of Australia
Reserve bank of New Zealand
Bank of Taiwan
Bank of Canada
Sveriges Riksbank
Swedish Riksbank
Central Bank of the Republic of Turkey
Central Bank of Brazil
0

200

400

600

800

1,000

1,200

Number of Mentions

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

1,400

1,600

1,800 2,000
Figure 7.4.6

175

C H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

Artificial IntelligenceIndex Report 2021

U.S. A I P O L I CY PA P E R S
What are the AI policy initiatives outside national andintergovernmental governments? We monitored 42
prominent organizations that deliver policy papers on
topics related to AI and assessed the primary topic aswell as the secondary topic on policy papers publishedin 2019 and 2020. (See the Appendix for a complete listof organizations included.) Those organizations areeither U.S.-based or have a sizable presence in the UnitedStates, and we grouped them into three categories: thinktanks, policy institutes and academia (27); civil society
organizations, associations and consortiums (9); andindustry and consultancy (6).
AI policy papers are defined as research papers, research
reports, blog posts, and briefs that focus on a specific policy
issue related to AI and provide clear recommendations

7.4 A I A N D
P O L I CY M A K I N G

for policymakers. Primary topics mean that such a topic is
the main focus of the policy paper, while secondary topics
mean that the policy paper either briefly touches on the
topic or the topic is a sub-focus of the paper.
Combined data for 2019 and 2020 suggests that the topics
of innovation and technology, international affairs andinternational security, and industry and regulation are
the main focuses of AI policy papers in the United States(Figure 7.4.7). Fewer documents placed a primary focus
on topics related to AI ethics—such as ethics, equity andinclusion; privacy, safety and security; and justice and law
enforcement—which have largely been secondary topics.
Moreover, topics bearing on the physical sciences, energy
and environment, humanities, and democracy havereceived the least attention in U.S. AI policy papers.

U.S. AI POLICY PRODUCTS by TOPIC, 2019-20 (SUM)
Source: Stanford HAI & AI Index, 2020 | Chart: 2021 AI Index ReportInnovation & Technology
Int'l Affairs & Int'l Security
Industry & Regulation
Workforce & Labor
Government & Public Administration
Privacy, Safety & Security
EthicsJustice & Law Enforcement
Equity & Inclusion
Education & Skills
Social & Behavioral Sciences
Health & Biological Sciences
Communications & Media
Secondary Topic

Democracy

Primary Topic

Humanities
Energy & Environment
Physical Sciences
0

20

40

60

80

100

120

140

160

Number of Policy Products
Figure 7.4.7

CHAPTER 7 PREVIEWTA B L E O F C O N T E N T S

176

Artificial IntelligenceIndex Report 2021

Appendix

APPENDIXTA B L E O F C O N T E N T S

Artificial IntelligenceIndex Report 2021

17 7

Artificial IntelligenceIndex Report 2021

APPENDIX

Appendix
CHAPTER 1

Research and Development

179

CHAPTER 2

Technical Performance

194

CHAPTER 3

The Economy

203

CHAPTER 4

AI Education

208

CHAPTER 5

Ethical Challenges of AI Applications

211

CHAPTER 6

Diversity in AI

214

CHAPTER 7

AI Policy and National Strategies

215

GLOBAL AI VIBRANCYAPPENDIXTA B L E O F C O N T E N T S

218

178

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

CHAPTER 1: RESEARCH & DEVELOPMENTELSEVIER
Prepared by Jörg Hellwig and Thomas A. Collins

SourceElsevier’s Scopus database of scholarly publications has
indexed more than 81 million peer-reviewed documents.
This data was compiled by Elsevier.

Methodology
Scopus tags its papers with keywords, publication dates,
country affiliations, and other bibliographic information.
The Elsevier AI Classifier leveraged the following features
extracted from the Scopus records that were returned asa result of querying against the provided approximately
800 AI search terms. Each record fed into the feature
creation also maintained a list of each search term that
hit for that particular record:• hasAbs: Boolean value whether or not the record had an
abstract text section in the record (e.g., some records
are only title and optional keywords)• coreCnt: number of core-scored search terms present
for the record• mediumCnt: number of medium-scored search terms
present for the record• lowCnt: number of low-scored search terms present for
the record• totalCnt: total number of search terms present for therecord• pcntCore: coreCnt/totalCnt• pcntMedium: mediumCnt/totalCnt• pcntLow: lowCnt/totalCnt• totalWeight = 5*coreCnt + 3*mediumCnt + 1*lowCnt• normWeight = if (has Abs) { totalWeight / (title.length +abstract.length) } else• { totalWeight/title.length}• hasASJC: Boolean value: does the record have anassociated ASJC list?
• isAiASJC: does ASJC list contain 1702?

APPENDIXTA B L E O F C O N T E N T S• isCompSciASJC does ASJC list contain a 17XX ASJC code(“1700,” “1701,” “1702,” “1703,” “1704,” “1705,” “1706,”“1707,” “1708,” “1709,” “1710,” “1711,” “1712”)• isCompSubj: does the Scopus record have a
ComputerScience subject code associated with it? Thisshould track 1:1 to isCompSciASJC. Scopus has 27 major
subject areas of which one is Computer Science. The
feature checks, if the publication is within Computer
Science or not. This is no exclusion.pcntCompSciASJC:
percentage of ASJC codes for record that are from the
CompSci ASJC code list
Details on Elsevier’s dataset defining AI, country
affiliations, and AI subcategories can be found in the 2018
AI Index Report Appendix.

Nuance• The Scopus system is retroactively updated. As a result,the number of papers for a given query may increaseover time.
• Members of the Elsevier team commented that data on
papers published after 1995 would be the most reliable.
The raw data has 1996 as the starting year for Scopus data.

Nuances specific to AI publications by region• Papers are counted utilizing whole counting rather
than fractional counting. Papers assigned to multiple
countries (or regions) due to collaborations are counted
toward each country (or region). This explains why topline numbers in a given year may not match individual
country numbers. For example, a paper assigned toGermany, France, and the United States will appear oneach country’s count, but only once for Europe (plusonce for the U.S.) as well as being counted only at the
global level.
• “Other” includes all other countries that have published
one or more AI papers on Scopus.

179

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Nuances specific to publications by topic• The 2017 AI Index Report showed only AI papers within
the CS category. In the 2018 and 2019 reports, all papers
tagged as AI were included, regardless of whether theyfell into the larger CS category.
• Scopus has a subject category called AI, which is asubset of CS, but this is relevant only for a subjectcategory approach to defining AI papers. The
methodology used for the report includes all papers,
since increasingly not all AI papers fall under CS.

Nuances specific to methodology• The entire data collection process was done by Elsevier
internally. The AI Index was not involved in the keyword
selection process or the counting of relevant papers.
• The boundaries of AI are difficult to establish, in part
because of the rapidly increasing applications inmany fields, such as speech recognition, computer
vision, robotics, cybersecurity, bioinformatics, andhealthcare. But limits are also difficult to define becauseof AI’s methodological dependency on many areas,such as logic, probability and statistics, optimization,
photogrammetry, neuroscience, and game theory—toname just a few. Given the community’s interest in AI
bibliometrics, it would be valuable if groups producing
these studies strived for a level of transparency in their
methods, which would support the reproducibilityof results, particularly on different underlying
bibliographic databases.

AI Training Set
A training set of approximately 1,500 publications
defines the AI field. The set is only the EID (the Scopus
identifier of the underlying publications). Publications
can be searched and downloaded either from Scopus
directly or via the API.The training set is a set of
publications randomly selected from the initial sevenmio publications. After running the algorithm we verify
the results of the training set with the gold set (expert
hand-checked publications which are definitely AI).

APPENDIXTA B L E O F C O N T E N T S

180

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

 I C R O S O F T ACA D E M I C G R A P H :M
M E T H O D O LO GY
Prepared by Zhihong Shen, Boya Xie, Chiyuan Huang,
Chieh-Han Wu, and Kuansan Wang

SourceThe Microsoft Academic Graph1 is a heterogeneous graphcontaining scientific publication records and citation
relationships between those publications, as well as
authors, institutions, journals, conferences, and fields ofstudy. This graph is used to power experiences in Bing,
Cortana, Word, and Microsoft Academic. The graph iscurrently being updated on a weekly basis. Learn more
about MAG here.

Methodology
MAG Data Attribution: Each paper is counted exactly
once. When a paper has multiple authors or regions,the credit is equally distributed to the unique regions.
For example, if a paper has two authors from theUnited States, one from China, and one from the United
Kingdom, then the United States, China, and the United
Kingdom each get one-third credit.
Metrics: Total number of published papers (journal
papers, conference papers, patents, repository2); total
number of citations of published papers.
Definition: The citation and reference countrepresents the number of respective metrics for AI
papers collected from all papers. For example, in“OutAiPaperCitationCountryPairByYearConf.csv,” a row
stating “China, United States, 2016, 14955” means that
China’s conference AI papers published in 2016 received
14,955 citations from (all) U.S. papers indexed by MAG.
Curating the MAG Dataset and References: Generally
speaking, the robots sit on top of a Bing crawler to readeverything from the web and have access to the entire
web index. As a result, MAG is able to program the robots

to conduct more web searches than a typical humancan complete. This helps disambiguate entities withthe same names. For example, for authors, MAG gets toadditionally use all the CVs and institutional homepages
on the web as signals to recognize and verify claims3.
MAG has found this approach to be superior to the resultsof the best of the KDD Cup 2013 competition, which uses
only data from within all publication records and Open
Researcher and Contributor Identifiers (ORCIDs).

Notes About the MAG Data
Conference Papers: After the contents and data sourceswere scrutinized, it was determined that some of the 2020
conference papers were not properly tagged with theirconference venue. Many conference papers in the MAG
system are under arXiv papers, but due to issues arisingfrom some data sources (including delays in DBLP andweb form changes on the ACM website), they were possibly
omitted as 2020 conference papers (ICML-PKDD, IROS, etc.).
However, the top AI conferences (selected not in terms of
publication count, but rather considering both publicationand citation count as well as community prestige) arecomplete. In 2020, the top 20 conferences presented 103,000
papers, which is 13.7% of all AI conference papers, and theyreceived 7.15 million citations collectively, contributing
47% of all citations received for all AI conference papers.
The number of 2020 conference publications is slightly
lower than in 2019. Data is known to be missing for ICCVand NAACL. About 100 Autonomous Agents and Multiagent
Systems (AAMAS) conference papers are erroneously
attributed to an eponymous journal.
Unknown Countries for Journals and Conferences:
For the past 20 to 30 years, 30% of journal and conference
affiliation data lacks affiliation by country or region, due
to errors in paper format, data source, and PDF parsing,among others.

1 See “A Review of Microsoft Academic Services for Science of Science Studies” and “Microsoft Academic Graph: When Experts Are Not Enough” for more details.
2 Repository as a publication type in MAG refers to both preprints and postprints. In the AI domain, it predominantly comes from arXiv. See “Is Preprint the Future of Science? A Thirty Year Journey of
Online Preprint Services” for details.
3 See “Machine Verification for Paper and Author Claims” and “How Microsoft Academic Uses Knowledge to Address the Problem of Conflation/Disambiguation” for details.

APPENDIXTA B L E O F C O N T E N T S

181

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXM I C R O S O F T ACA D E M I C G R A P H :PAT E N T DATA C H A L L E N G E
As mentioned in the report, the patent data—especiallythe affiliation information—is incomplete in the MAG
database. The reason for the low coverage is twofold.
First, applications published by the patent offices often
identify the inventors by their residencies not affiliations.
While patent applications often have the information
about the “assignees” of a patent, they do not necessarilymean the underlying inventions originate from theassignee institutions. Therefore, detected affiliations may
be inaccurate. In case a patent discloses the scholarly
publications underlying the invention, MAG can infer
inventors’ affiliations through the scholarly publications.
Second, to maximize intellectual property protection
around the globe, institutions typically file multiple
patent applications on the same invention under various

jurisdictions. These multiple filings, while appear very
different because the titles and inventor names are oftentranslated into local languages, are in fact the result of asingle invention. Raw patent counts therefore inflate theinventions in their respective domains. To remediate thisissue, MAG uses the patent family ID feature to combine allfilings with the original filing, which allows the database
to count filings all around the world of the same origin
only once.4 Conflating the multiple patent applications of
the same invention is not perfect, and over-conflations ofpatents are more noticeable in MAG than scholarly articles.
These challenges raise questions about the reliability ofdata on the share of AI patent publications by both region
and geographic area. Those charts are included below.

By Region
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20

AI Patent Publications (% of World Total)

Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

8%

6%

4%
3.2%, North America
2.6%, East Asia & Pacific2%

1.6%, Europe & Central Asia
0.1%, South Asia
0.0%, Sub-Saharan Africa
0.0%, Latin America & Caribbean
0.0%, Middle East & North Africa

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.1

4 Read “Sharpening Insights into the Innovation Landscape with a New Approach to Patents” for more details.

APPENDIXTA B L E O F C O N T E N T S

182

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

By Geographic Area
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of World Total)

8%

6%

4%
3.2% US2%

1.3% EU
0.4% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.2

Citation
AI PATENT CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Citations (% of World Total)

25%

20%

15%

10%

8.6% US5%2.1% EU
2.0% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.3

APPENDIXTA B L E O F C O N T E N T S

183

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

M I C R O S O F T ACA D E M I C G R A P H :MEASUREMENT CHALLENGES AND
A LT E R N AT I V E D E F I N I T I O N O F A I
As the AI Index team discussed in the paper “Measurementin AI Policy: Opportunities and Challenges,” choosing how
to define AI and correctly capture relevant bibliometricdata remain challenging. Data in the main report is basedon a restricted definition of AI, adopted by MAG, that aligns
with what has been used in previous AI Index reports.
One consequence is that such a definition excludes many
AI publications from venues considered to be core AI
venues. For example, only 25% of conference publicationsin the 2020 AAAI conference are included in the original
conference dataset.

papers in the MAG database tagged with a field of study
that is categorized in either the “artificial intelligence”
or the “machine learning” field of study as well as theirsubtopics in the MAG taxonomy.5 This is a more liberal
definition than the one used by MAG, which considers only
those publications tagged with “artificial intelligence”
as AI publications. For example, an application paper inbiology that uses ML techniques will be counted as an AI
publication under the OECD definition, but not under the
MAG definition unless the paper is specifically tagged in
the AI category.

To spur discussion on this important topic, this section
presents the MAG data with an alternative definition of AI
used by the Organisation for Economic Co-operation andDevelopment (OECD). OECD defines AI publications as

Charts corresponding to those in the main text but using
the OECD definition are presented below. The overall
trends are very similar.

AI Journal Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Journal Publications (in Thousands)

300

250

200

150

100

50

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.1a

5 Read the OECD.AI Policy Observatory MAG methodological note for more details on the MAG-OECD definition of AI and “A Web-scale System for Scientific Knowledge Exploration” on the MAG Taxonomy.

APPENDIXTA B L E O F C O N T E N T S

184

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
7.9%

AI Journal Publications (% of All Publications)

8%

6%

4%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2%

Figure 1.5.1b

OECD DEFINITION: AI JOURNAL PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%
16.5% Europe & Central Asia
16.2% North America

15%

10%

5%

4.2% South Asia
4.1% Middle East & North Africa

2025

2024

2023

2022

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2021

2.2% Latin America & Caribbean
1.0% Sub-Saharan Africa

0%

2020

AI Journal Publications (% of World Total)

23.0% East Asia & PacificFigure 1.5.2

APPENDIXTA B L E O F C O N T E N T S

185

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI JOURNAL PUBLICATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Publications (% of World Total)

20%

14.4%
China

15%

14.1%
US

10%

10.8%
EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.3

OECD DEFINITION: AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Publications (% of World Total)

20%

15%

10%

8.4%
China
8.2%
US
5.8%
EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.4

APPENDIXTA B L E O F C O N T E N T S

186

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

AI Conference Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of Publications (in Thousands)

100

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.5aOECD DEFINITION: AI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of All Publications)

40%

37.6%

35%

30%

25%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

20%

Figure 1.5.5b

APPENDIXTA B L E O F C O N T E N T S

187

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXOECD DEFINITION: AI CONFERENCE PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of World Total)

40%

30%
26.5% East Asia & Pacific21.2% North America

20%

20.0% Europe & Central Asia

10%5.0% South Asia
2.4% Middle East & North Africa
1.8% Latin America & Caribbean

0%

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.4% Sub-Saharan Africa

Figure 1.5.6

OECD DEFINITION: AI CONFERENCE PUBLICATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of World Total)

25%

18.9%
US

20%

15.0%
China

15%
14.2%
EU

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.7

APPENDIXTA B L E O F C O N T E N T S

188

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI CONFERENCE CITATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

14.3%
US

AI Conference Citation (% of World Total)

15%

10%

4.2%
China

5%

4.2%
EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.8

AI Patent Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI PATENT PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Patent Publications (in Thousands)

200

150

100

50

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.9a

APPENDIXTA B L E O F C O N T E N T S

189

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXOECD DEFINITION: AI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of All Publications)

5%

4.6%

4%

3%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2%

Figure 1.5.9b

OECD DEFINITION: AI PATENT PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

8%

6%

4%
3.3% North America
2.1% East Asia & Pacific2%

1.4% Europe & Central Asia
0.1% Middle East & North Africa
0.1% South Asia
0.0% Sub-Saharan Africa

0%

2024

2023

2022

2021

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.0% Latin America & Caribbean

2020

AI Patent Publications (% of World Total)

10%

Figure 1.5.10

APPENDIXTA B L E O F C O N T E N T S

190

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI PATENT PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of World Total)

10%

8%

6%

4%

3.2% US2%
1.0% EU
0.4% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.11

OECD DEFINITION: AI PATENT CITATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Citation (% of World Total)

25%

20%

15%

10%
5.6% US5%
1.2% China
1.1% EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.12

APPENDIXTA B L E O F C O N T E N T S

191

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

PA P E R S O N A R X I V
Prepared by Jim Entwood and Eleonora Presani

SourcearXiv.org is an online archive of research articles in
the fields of physics, mathematics, computer science,quantitative biology, quantitative finance, statistics,
electrical engineering and systems science, andeconomics. arXiv is owned and operated by Cornell
University. See more information on arXiv.org.

Methodology
Raw data for our analysis was provided byrepresentatives at arXiv.org. The keywords we selected,
and their respective categories, are below:
Artificial intelligence (cs.AI)
Computation and language (cs.CL)
Computer vision and pattern recognition (cs.CV)
Machine learning (cs.LG)
Neural and evolutionary computing (cs.NE)
Robotics (cs.RO)
Machine learning in stats (stats.ML)

arXiv is actively looking at ways to improve how it canbetter support AI/ML researchers as the field grows
and discovering content becomes more challenging.
For example, there may be ways to create finer grained
categories in arXiv for machine learning to help
researchers in subfields share and find work more easily.
The other rapidly expanding area is computer vision,
where there is considerable overlap for ML applicationsof computer vision.

Nuance• Categories are self-identified by authors—those shown
are selected as the “primary” category. Thus there is nota single automated categorization process. Additionally,
the artificial intelligence or machine learning categories
may be categorized by other subfields or keywords.
• arXiv team members suggest that participation onarXiv can breed greater participation, meaning that anincrease in a subcategory on arXiv could drive overindexed participation by certain communities.

For most categories, arXiv provided data for 2015–2020.
To review other categories’ submission rates on arXiv, seearXiv.org’s submission statistics.
The arXiv team has been expanding the publicly available
submission statistics. This is a tableau-based application
with tabs at the top for various displays of submission
stats and filters on the side bar to drill down by topic.
(Hover over the charts to view individual categories.) The
data is meant to be displayed on a monthly basis withdownload options.

APPENDIXTA B L E O F C O N T E N T S

192

Artificial IntelligenceIndex Report 2021

N E S TA

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

G I T H U B S TA R S

Prepared by Joel Kliger and Juan Mateos-Garcia

SourceSource

GitHub: star-history (available at star history website)
was used to retrieve the data.

Details can be found in the following publication:
Deep Learning, Deep Change? Mapping the Developmentof the Artificial Intelligence General Purpose Technology

Methodology
Deep learning papers were identified through a topic
modeling analysis of the abstracts of arXiv papers in
the CS (computer science) and stats.ML (statistics:
machine learning category) arXiv categories. The data
was enriched with institutional affiliation and geographic
information from the Microsoft Academic Graph andthe Global Research Identifier. Nesta’s arXlive tool isavailable here.

Access the CodeThe code for data collection and processing can be found
here; or, without the infrastructure overhead here.

APPENDIXTA B L E O F C O N T E N T S

Methodology
The visual in the report shows the number of stars forvarious GitHub repositories over time. The repositories
include the following:
apache/incubator-mxnet, BVLC/cafe, cafe2/cafe2, dmlc/
mxnet, fchollet/keras, Microsoft/CNTK, pytorch/pytorch,scikit-learn/scikit-learn, tensorflow/tensorflow, Theano/
Theano, Torch/Torch7.

NuanceThe GitHub Archive currently does not provide a way
to count when users remove a star from a repository.
Therefore, the reported data slightly overestimates thenumber of stars. A comparison with the actual numberof stars for the repositories on GitHub reveals that the
numbers are fairly close and that the trends remainunchanged.

193

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCECHAPTER 2: TECHNICAL PERFORMANCEI M AG E N E T: AC C U R ACY
Prepared by Jörg Hellwig and Thomas A. Collins

SourceData on ImageNet accuracy was retrieved through an arXiv
literature review. All results reported were tested on the
LSRVC 2012 validation set, as the results on the test set,
which are not significantly different, are not public. Theirordering may differ from the results reported on the LSRVC
website, since those results were obtained on the test set.
Dates we report correspond to the day when a paper was
first published to arXiv, and top-1 accuracy corresponds
to the result reported in the most recent version of each
paper. We selected a top result at any given point intime from 2012 to Nov. 17, 2019. Some of the results wemention were submitted to LSRVC competitions over theyears. Image classification was part of LSRVC through2014; in 2015, it was replaced with an object localizationtask, where results for classification were still reported but
no longer a part of the competition, having instead been
replaced by more difficult tasks.

For papers published in 2014 and later, we report the best
result obtained using a single model (we did not include
ensembles) and using single-crop testing. For the threeearliest models (AlexNet, ZFNet, Five Base), we reported
the results for ensembles of models.
While we report the results as described above, due
to the diversity in models, evaluation methods, andaccuracy metrics, there are many other ways to report
ImageNet performance. Some possible choices include:• Evaluation set: validation set (available publicly) or testset (available only to LSRVC organizers)• Performance metric: Top-1 accuracy (whether thecorrect label was the same as the first predicted label
for each image) or top-5 accuracy (whether the correctlabel was present among the top five predicted labels
for each image)• Evaluation method: single-crop or multi-crop

To highlight progress here in top-5 accuracy, we have taken scores from the following papers, without extra training data:
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Adversarial Examples Improve Image Recognition
OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks
Local Relation Networks for Image Recognition
Densely Connected Convolutional Networks
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Squeeze-and-Excitation Networks
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
MultiGrain: A Unified Image Embedding for Classes and Instances
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Billion-Scale Semi-Supervised Learning for Image Classification
GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
RandAugment: Practical Data Augmentation with No Separate Search
Fixing the Train-Rest Resolution Discrepancy

APPENDIXTA B L E O F C O N T E N T S

194

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCE

To highlight progress here in top-5 accuracy, we have taken scores from the following papers, with extra training data:
Meta Pseudo Labels
Self-Training with Noisy Student Improves ImageNet Classification
Big Transfer (BiT): General Visual Representation Learning
ImageNet Classification with Deep Convolutional Neural Networks
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Xception: Deep Learning with Depthwise Separable Convolutions
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Self-training with Noisy Student Improves ImageNet Classification
To highlight progress here in top-1 accuracy, we have taken scores from the following papers, without extra training data:
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Adversarial Examples Improve Image Recognition
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Densely Connected Convolutional Networks
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Dual Path Networks
Res2Net: A New Multi-Scale Backbone Architecture
Billion-Scale Semi-Supervised Learning for Image Classification
Squeeze-and-Excitation Networks
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
MultiGrain: A Unified Image Embedding for Classes and Instances
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Billion-Scale Semi-Supervised Learning for Image Classification
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
RandAugment: Practical Data Augmentation with No Separate Search
Fixing the Train-Test Resolution Discrepancy
To highlight progress here in top-1 accuracy, we have taken scores from the following papers, without extra training data:
Meta Pseudo Labels
Sharpness-Aware Minimization for Efficiently Improving Generalization
An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale
Fixing the Train-Test Resolution Discrepancy: FixEfficientNet
Self-training with Noisy Student Improves ImageNet Classification
Big Transfer (BiT): General Visual Representation Learning
ImageNet Classification with Deep Convolutional Neural Networks
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Xception: Deep Learning with Depthwise Separable Convolutions
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Self-training with Noisy Student Improves ImageNet Classification
The estimate of human-level performance is from Russakovsky et al, 2015. Learn more about the LSVRC ImageNet
competition and the ImageNet data set.

APPENDIXTA B L E O F C O N T E N T S

195

Artificial IntelligenceIndex Report 2021

I M AG E N E T: T R A I N I N G T I M E
Trends can also be observed by studying research papers
that discuss the time it takes to train ImageNet on any
infrastructure. To gather this data, we looked at research
papers from the past few years that tried to optimize
for training ImageNet to a standard accuracy level while
competing on reducing the overall training time.

SourceThe data is sourced from MLPerf. Detailed data for runs for
specific years are available:
2020: MLPerf Training v0.7 Results
2019: MLPerf Training v0.6 Results
2018: MLPerf Training v0.5 Results

Notes
Data from MLPerf is available in cloud systems for rent.
Available On Premise systems contain only componentsthat are available for purchase. Preview systems must be
submittable as Available In Cloud or Available on Premisein the next submission round. Research, Development, orInternal (RDI) contain experimental, in development, orinternal-use hardware or software. Each row in the results
table is a set of results produced by a single submitter using the same software stack and hardware platform. Eachrow contains the following information:
Submitter: the organization that submitted the results
System: general system description
Processor and count: the type and number of CPUs used, if
CPUs perform the majority of ML compute
Accelerator and count: the type and number of accelerators used, if accelerators perform the majority of ML
computeSoftware: the ML framework and primary ML hardware
library used
Benchmark results: training time to reach a specified target quality, measured in minutes
Details: link to metadata for submission
Code: link to code for submission
Notes: arbitrary notes from the submitter

APPENDIXTA B L E O F C O N T E N T S

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCEI M AG E N E T: T R A I N I N G C O S T

SourceDAWNBench is a benchmark suite for end-to-end,
deep-learning training and inference. Computation
time and cost are critical resources in building deepmodels, yet many existing benchmarks focus solely onmodel accuracy. DAWNBench provides a reference setof common deep-learning workloads for quantifying
training time, training cost, inference latency, andinference cost across different optimization strategies,
model architectures, software frameworks, clouds, andhardware. More details available at DawnBench.

Note
The DawnBench data source has been deprecated for theperiod after March 2020, and MLPerf is the most reliable
and updated source for AI compute measurements.

C O C O : K E Y P O I N T D E T E CT I O N
The data for COCO keypoint detection data is sourced from
COCO keypoints leaderboard.

C O C O : D E N S E P O S E E S T I M AT I O N
We gathered data from the CODALab 2020 challenge andread arXiv repository papers to build comprehensive dataon technical progress in this challenge. The detailed list of
papers and sources used in our survey include:
DensePose: Dense Human Pose Estimation In the Wild
COCO-DensePose 2018 CodaLab
Parsing R-CNN for Instance-Level Human Analysis
Capture Dense: Markerless Motion Capture Meets Dense
Pose Estimation
Slim DensePose: Thrifty Learning from Sparse Annotations
and Motion Cues
COCO-DensePose 2020 CodaLab
Transferring Dense Pose to Proximal Animal Classes
Making DensePose Fast and Light
SimPose: Effectively Learning DensePose and Surface
Normals of People from Simulated Data

196

Artificial IntelligenceIndex Report 2021

AC T I V I T Y N E T: T E M P O R A L
LO CA L I Z AT I O N TA S K
In the challenge, there are three separate tasks, butthey focus on the main problem of temporally localizing
where activities happen in untrimmed videos fromthe ActivityNet benchmark. We have compiled several
attributes for the task of temporal localization at thechallenge over the last four rounds. Below is a link to
the overall stats and trends for this task, as well as somedetailed analysis (e.g., how has the performance forindividual activity classes improved over the years? Whichare the hardest and easiest classes now? Which classes
have the most improvement over the years?). See the
Performance Diagnosis (2020) tab for a detailed trends
update. Please see ActivityNet Statisticsin the public data
folder for more details.

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCEYO LO ( YO U O N LY LO O K O N C E )YOLO is a neural network model mainly used for thedetection of objects in images and in real-time videos.
mAP (mean average precision) is a metric that is used
to measure the accuracy of object detectors. It is acombination of precision and recall. mAP is the average ofthe precision and recall calculated over a document. The
performance of YOLO has increased gradually with thedevelopment of new architectures and versions in past
years. With the increase in size of model, its mean average
precision increases as well, with a corresponding decrease
in FPS of the video.
We conducted a detailed survey of arXiv papers and
GitHub repository to segment progress in YOLO acrossits various versions. Below are the references for original
sources:
YOLOv1:You Only Look Once: Unified, Real-Time Object Detection  
YOLOv2:
YOLO9000: Better, Faster, Stronger
YOLO: Real-Time Object Detection  
YOLOv3:
YOLOv3: An Incremental Improvement
Learning Spatial Fusion for Single-Shot Object Detection
GitHub: ultralytics/yolov3
YOLOv4:
YOLOv4: Optimal Speed and Accuracy of Object Detection
GitHub: AlexeyAB/darknet
YOLOv5:
GitHub: ultralytics/yolov5
PP-YOLO:
PP-YOLO: An Effective and Efficient Implementation of
Object Detector
POLY-YOLO:
Poly-YOLO: Higher Speed, More Precise Detection andInstance Segmentation for YOLOV3APPENDIXTA B L E O F C O N T E N T S

197

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCEV I S UA L Q U E S T I O N A N S W E R I N G( VQ A )PA P E R S W I T H C O D E :PA P E R A N D C O D E L I N K I N G

VQA accuracy data was provided by the VQA team. Learn
more about VQA here. More details on VQA 2020 are
available here.

We used paperswithcode (PWC) for referencing technicalprogress where available. Learn more about PWC here andsee the public link here.

Methodology
Given an image and a natural language question about theimage, the task is to provide an accurate natural language
answer. The challenge is hosted on the VQA Challenge
website. The challenge is hosted on EvalAI. The challenge
link is here.
The VQA v2.0 training, validation, and test sets, containing
more than 250,000 images and 1.1 million questions,are available on the download page. All questions areannotated with 10 concise, open-ended answers each.
Annotations on the training and validation sets are
publicly available.
VQA Challenge 2020 is the fifth edition of the VQA
Challenge. Results from previous versions of the VQA
Challenge were announced at the VQA Challenge
Workshop in CVPR 2019, CVPR 2018, CVPR 2017, and CVPR
2016. More details about past challenges can be foundhere: VQA Challenge 2019, VQA Challenge 2018, VQA
Challenge 2017, VQA Challenge 2016.  
VQA had 10 humans answer each question. More details
about the VQA evaluation metric and human accuracy
can be found here (see Evaluation Code section) andin sections three (“Answers”) and four (“Inter-Human
Agreement”) of the paper.  
See slide 56 for the progress graph in VQA in the 2020
Challenge. The values corresponding to the progress graph
are available in a sheet. Here is the information about theteams that participated in the 2020 challenge and theiraccuracies. For more details about the teams, please referto the VQA website.

APPENDIXTA B L E O F C O N T E N T S

Methodology
For papers, we follow specific ML-related categories onarxiv (see [1] below for the full list) and the major ML
conferences (NeurIPS, ICML, ICLR, etc.). For code, we
follow GitHub repositories mentioning papers. We havegood coverage of core ML topics but are missing some
applications—for instance, applications of ML in medicine
or bioinformatics, which are usually in journals behind
paywalls. For code, the dataset is fairly unbiased (as longas the paper is freely available).
For tasks (e.g., “image classification”), the dataset hasannotated those on 1,600 state-of-the-art papers from thedatabase, published in 2018 Q3.
For state-of-the-art tables (e.g., “image classification on
ImageNet”), the data has been scraped from different
sources (see the full list here), and a large number focusing
on CV and NLP were hand-annotated. A significant portionof our data was contributed by users, and they have added
data based on their own preferences and interests. Arxiv
categories we follow:
ARXIV_CATEGORIES = “cs.CV”, “cs.AI”, “cs.LG”, “cs.CL”, “cs.
NE”, “stat.ML”,”cs.IR”}

Process of Extracting Dataset at Scale1) Follow various paper sources (as described above) for
new papers.
2) Conduct a number of predefined searches on GitHub(e.g., for READMEs containing links to arxiv).
3) Extract GitHub links from papers.
4) Extract paper links from GitHub.
5) Run validation tests to decide if links from 3) and 4) arebona fide links or false positives.
6) Let the community fix any errors and/or add any missing
values.

198

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCENIST FRVT

SourceThere are two FRVT evaluation leaderboards available here: 1:1 Verification and 1:N Identification

Nuances about FRVT evaluation metrics
Wild Photos have some identity labeling errors as the best algorithm has a low false non-match rate (FNMR), but
obtaining complete convergence is difficult. This task will be retired in the future. The data became public in 2018 andhas become easier over time. Wild is coming from public web sources. So it is possible those same images have been
scrapped from the web by developers. There is no training in the FRVT data, only test data.
The 1:1 and 1:N should be studied separately. The differences include algorithmic approaches, particularly fast searchalgorithms are especially useful in 1:N whereas speed is not a factor in 1:1.

S U P E R G LU E
The SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about the SuperGLUE benchmark
are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks and evaluation metrics for SuperGLUE are:NAME

IDENTIFIERMETRIC

Broad Coverage Diagnostics

AX-b

Matthew’s Corr

CommitmentBank

CB

Avg. F1 / Accuracy

Choice of Plausible Alternatives

COPA

Accuracy

Multi-Sentence Reading Comprehension

MultiRC

F1a / EM

Recognizing Textual Entailment

RTE

Accuracy

Words in Context

WiC

Accuracy

The Winograd Schema Challenge

WSC

Accuracy

BoolQ

BoolQ

Accuracy

Reading Comprehension with Commonsense Reasoning

ReCoRDF1 / Accuracy

Winogender Schema Diagnostics

AX-g

Gender Parity / Accuracy

V I S UA L C O M M O N S E N S E R E A S O N I N G ( VC R )Technical progress for VCR is taken from the VCR leaderboard. VCR has two different subtasks:• Question Answering (Q->A): A model is provided a question and has to pick the best answer out of four choices. Only
one of the four is correct.
• Answer Justification (QA->R): A model is provided a question, along with the correct answer, and it must justify it bypicking the best rationale among four choices.
The two parts with the Q->AR metrics are combined in which a model only gets a question right if it answers correctly
and picks the right rationale. Models are evaluated in terms of accuracy (%).

APPENDIXTA B L E O F C O N T E N T S

199

Artificial IntelligenceIndex Report 2021

VOXC E L E B
VoxCeleb is an audio-visual dataset consisting of shortclips of human speech, extracted from interview videos
uploaded to YouTube. VoxCeleb contains speech from
7,000-plus speakers spanning a wide range of ethnicities,
accents, professions, and ages—amounting to over a
million utterances (face-tracks are captured “in the wild,”with background chatter, laughter, overlapping speech,
pose variation, and different lighting conditions) recorded
over a period of 2,000 hours (both audio and video). Eachsegment is at least three seconds long. The data contains
an audio dataset based on celebrity voices, shorts,
films, and conversational pieces (e.g., talk shows). The
initial VoxCeleb 1 (100,000 utterances taken from 1,251
celebrities on YouTube) was expanded to VoxCeleb 2 (1million utterances from 6,112 celebrities).
However, in earlier years of the challenge, top-1 and top-5
scores were also reported. For top-1 score, the system iscorrect if the target label is the class to which it assigns
the highest probability. For top-5 score, the system iscorrect if the target label is one of the five predictions
with the highest probabilities. In both cases, the top scoreis computed as the number of times a predicted label
matches the target label, divided by the number of data
points evaluated.
The data is extracted from different years of thesubmission challenges, including:• 2 017: VoxCeleb: A Large-Scale Speaker Identification Dataset• 2018: VoxCeleb2: Deep Speaker Recognition• 2 019: Voxceleb: Large-Scale Speaker Verification in the Wild
• 2020: Query ExpansionSystem for the VoxCeleb Speaker
Recognition Challenge 2020

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCEB O O L E A N SAT I S F I A B I L I T Y
PROBLEM
Analysis and text by Lars Kotthoff

Primary Source and Data Sets
The Boolean Satisfiability Problem (SAT) determines
whether there is an assignment of values to a set ofBoolean variables joined by logical connectives thatmakes the logical formula it represents true. SAT was thefirst problem to be proven NP-complete, and the firstalgorithms to solve it were developed in the 1960s. Many
real-world problems, such as circuit design, automatedtheorem proving, and scheduling, can be represented andsolved efficiently as SAT. The annual SAT competition isdesigned to present a snapshot of the state-of-the-art andhas been running for almost 20 years.
We took the top-ranked, median-ranked, and bottomranked solvers from each of the last five years (2016-2020)of the SAT competition. We ran all 15 solvers on all 400 SAT
instances from the main track of the 2020 competition. Moreinformation on the competition, as well as the solvers andinstances, is available at the SAT competition website.

Results
We ran each solver on each instance on the same
hardware, with a time limit of 5,000 CPU seconds per
instance, and measured the time it took a solver to solve
an instance in CPU seconds. Ranked solvers always returncorrect results, hence we do not consider correctness
as a metric. Except for the 2020 competition solvers,
we evaluated the performance of the SAT solvers on aset of instances different from the set of instances they
competed on. Further, our hardware is different from what
was used for the SAT competition. The results we report
here will therefore differ from the exact results reported forthe respective SAT competitions.
The Shapley value is a concept from cooperative gametheory that assigns a contribution to the total value thata coalition generates to each player. It quantifies how
important each player is for the coalition and has several
desirable properties that make the distribution of thetotal value to the individual players fair. For example,

APPENDIXTA B L E O F C O N T E N T S

200

Artificial IntelligenceIndex Report 2021

the Shapley value is used to distribute airport costs to its
users, allocate funds to different marketing campaigns,
and in machine learning, where it helps render complex
black-box models more explainable.
In our context, it quantifies the contribution of a solver
to the state-of-the-art through the average performance
improvement it provides over a set of other solvers and overall subsets of solvers (Fréchette et al. (2016)). For a given setof solvers, we choose the respective best for each instanceto solve. By including another solver and being able tochoose it, overall solving performance improves, with thedifference to the original set of solvers being the marginalcontribution of the added solver. The average marginalcontribution to all sets of solvers is the Shapley value.
Quantifying the contribution of a solver through the
Shapley value compares solvers from earlier competitions
to solvers in later competitions. This is often not a faircomparison, as later solvers are often improved versionsof earlier solvers, and the contribution of the solver to the
future state-of-the-art will always be low. The temporal
Shapley value (Kotthoff et al. (2018)) solves this problem
by considering the time a particular solver was introduced
when quantifying its contribution to the state-of-the-art.

AU TO M AT E D T H E O R E M P R OV I N G
Analysis and text by Christian Suttner, Geoff Sutcliffe, andRaymond Perrault

1. Motivation
Automated Theorem Proving (ATP) (also referred to
as Automated Deduction) is a subfield of automatedreasoning, concerned with the development and use of
systems that automate sound reasoning: the derivationof conclusions that follow inevitably from facts. ATP
systems are at the heart of many computational tasks andare used commercially, e.g., for integrated circuit designand computer program verification. ATP problems are
typically solved by showing that a conjecture is or is nota logical consequence of a set of axioms. ATP problems
are encoded in a chosen logic, and an ATP system for

APPENDIXTA B L E O F C O N T E N T S

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCE

that logic is used to (attempt to) solve the problem. A
key concern of ATP research is the development of more
powerful systems, capable of solving more difficult
problems within the same resource limits. In order toassess the merits of new techniques, sound empiricalevaluations of ATP systems are key.

2. Analysis
For the evaluation of ATP systems, there exists a large andgrowing collection of problems called the TPTP problem
library. The current release v7.4.0 (released June 10,2020) contains 23,291 ATP problems, structured into 54
topic domains (e.g., Set Theory, Software Verification,
Philosophy, etc.). Orthogonally, the TPTP is divided into
Specialist Problem Classes (SPCs), each of which contains
problems with a specified set of logical, language, andsyntactic characteristics (e.g. first-order logic theorems
with some use of equality). The SPCs allow ATP systemdevelopers to select problems and evaluate theirsystems appropriately. Since its first release in 1993,
many researchers have used the TPTP as an appropriate
and convenient basis for ATP system evaluation. Overthe years, the TPTP has also increasingly been used asa conduit for ATP users to contribute samples of theirproblems to ATP system developers. This exposes the
problems to ATP system developers, who can thenimprove their systems’ performances on the problems,
which completes a cycle to provide users with more
effective tools.
Associated with the TPTP is the TSTP solution library,which maintains updated results from running all current
versions of ATP systems (available to the maintainer) on
all the TPTP problems. One use of the TSTP is to compute
TPTP problem difficulty ratings: Easy problems, which aresolved by all ATP systems, have a rating of 0.0; difficult
problems, which are solved by some ATP systems, haveratings between 0.0 and 1.0; unsolved problems, which are
not solved by any ATP system, have a rating of 1.0. Notethat the rating for a problem is not strictly decreasing, asdifferent ATP systems and versions become available forpopulating the TSTP. The history of each TPTP problem’s

201

Artificial IntelligenceIndex Report 2021

ratings is saved with the problem, which makes it possible
to tell when the problem was first solved by any ATP
system (the point at which its rating dropped below
1.0). That information has been used here to obtain an
indication of progress in the field.
The simplest way to measure progress takes a fixed set ofproblems that has been available (and unchanged) in the
TPTP from some chosen initial TPTP release, and then for
the TPTP releases from then on, counts how many of the
problems had been solved from that release. The analysis
reports the fraction of problems solved for each release.
This simple approach is unambiguous, but it does not take
into account new problems that are added to the TPTP
after the initial release.

APPENDIXC H A P T E R 2:
T E C H N I CA LPERFORMANCEThe table here in the public data folder shows thebreakdown of TPTP problems by content fields, as well
as by SPCs used in the analysis. The totals are slightly
larger than those shown in the analysis, as someproblems were left out for technical reasons (no scoresavailable, problems revised over time, etc.).

The analysis used here extends the “Fixed Set” analysis,taking into account new problems added after the initial
release. As it is not possible to run all previously available
ATP systems on new problems when they are added, thisapproach assumes that if a problem is unsolved by current
ATP systems when it is added to the TPTP, then it would
have been unsolved by previously available ATP systems.
Under that assumption, the new problem is retrospectively“added” to prior TPTP releases for the analysis. If a
problem is solved when it is added to the TPTP, it isignored because it may have been solved in prior versions
as well, and therefore should not serve as an indicator ofprogress. This analysis reports the fraction of problems
solved for each release, but note that the fraction is withrespect to both the number of problems actually in therelease and also the problems retrospectively “added.”
The growing set analysis is performed on the whole TPTP
and on four SPCs. These were chosen because many ATP
problems in those forms have been contributed to the
TPTP, and correspondingly there are many ATP systems
that can attempt them; they represent the “real world”
demand for ATP capability.

APPENDIXTA B L E O F C O N T E N T S

202

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 3:
ECONOMYCHAPTER 3: ECONOMY
LINKEDIN
Prepared by Mar Carpanelli, Ramanujam MV, and Nathan
Williams

Country Sample
Included countries represent a select sample of eligible
countries with at least 40% labor force coverage byLinkedIn and at least 10 AI hires in any given month.
China and India were included in this sample because of
their increasing importance in the global economy, but
LinkedIn coverage in these countries does not reach 40%of the workforce. Insights for these countries may notprovide as full a picture as other countries, and should beinterpreted accordingly.

SkillsLinkedIn members self-report their skills on their LinkedIn
profiles. Currently, more than 35,000 distinct, standardized
skills are identified by LinkedIn. These have been codedand classified by taxonomists at LinkedIn into 249 skill
groupings, which are the skill groups represented in thedataset. The top skills that make up the AI skill grouping
are machine learning, natural language processing, data
structures, artificial intelligence, computer vision, image
processing, deep learning, TensorFlow, Pandas (software),and OpenCV, among others.
Skill groupings are derived by expert taxonomists througha similarity-index methodology that measures skill
composition at the industry level. Industries are classified
according to the ISIC 4 industry classification (Zhu et al.,2018).

AI Skills Penetration
The aim of this indicator is to measure the intensity of AI
skills in an entity (in a particular country, industry, gender,etc.) through the following methodology:• Compute frequencies for all self-added skills by LinkedIn
members in a given entity (occupation, industry, etc.) in
2015–2020.

APPENDIXTA B L E O F C O N T E N T S• Re-weight skill frequencies using a TF-IDF model to get
the top 50 most representative skills in that entity. These
50 skills compose the “skill genome” of that entity.
• Compute the share of skills that belong to the AI skill
group out of the top skills in the selected entity.
Interpretation: The AI skill penetration rate signals theprevalence of AI skills across occupations, or the intensitywith which LinkedIn members utilize AI skills in theirjobs. For example, the top 50 skills for the occupation ofengineer are calculated based on the weighted frequency
with which they appear in LinkedIn members’ profiles. If
four of the skills that engineers possess belong to the AI
skill group, this measure indicates that the penetration of
AI skills is estimated to be 8% among engineers (e.g., 4/50).

Relative AI Skills Penetration
To allow for skills penetration comparisons acrosscountries, the skills genomes are calculated and a relevantbenchmark is selected (e.g., global average). A ratio is thenconstructed between a country’s and the benchmark’s AI
skills penetrations, controlling for occupations.
Interpretation: A country’s relative AI skills penetrationof 1.5 indicates that AI skills are 1.5 times as frequent as in
the benchmark, for an overlapping set of occupations.

Global Comparison
For cross-country comparison, we present the relative
penetration rate of AI skills, measured as the sum of thepenetration of each AI skill across occupations in a given
country, divided by the average global penetration of AI
skills across the overlapping occupations in a sample ofcountries.
Interpretation: A relative penetration rate of 2 meansthat the average penetration of AI skills in that country
is two times the global average across the same set ofoccupations.

203

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 3:
ECONOMY

Global Comparison: By Industry

B U R N I N G G L A S S T E C H N O LO G I E S

The relative AI skills penetration by country for industry
provides an in-depth sectoral decomposition of AI skill
penetration across industries and sample countries.

Prepared by Bledi Taska, Layla O’Kane, and Zhou Zhou

Interpretation: A country’s relative AI skill penetration
rate of 2 in the education sector means that the average
penetration of AI skills in that country is two times theglobal average across the same set of occupations in that
sector.

LinkedIn AI Hiring Index
The LinkedIn AI hiring rate is calculated as the total
number of LinkedIn members who are identified as AI
talent and added a new employer in the same month the
new job began, divided by the total number of LinkedIn
members in the country. By analyzing only the timeliest
data, it is possible to make month-to-month comparisonsand account for any potential lags in members updating
their profiles.
The baseline time period is typically a year, and it isindexed to the average month/period of interest duringthat year. The AI hiring rate is indexed against the averageannual hiring in 2016; for example, an index of 3.5 for
Brazil in 2020 indicates that the AI hiring rate is 3.5 times
higher in 2020 than the average in 2016.
Interpretation: The hiring index means the rate ofhiring in the AI field, specifically how fast each country is
experiencing growth in AI hiring.

Top AI Skills
AI skills most frequently added by members during
2015–2020 period.

Burning Glass Technologies delivers job market analytics
that empower employers, workers, and educators tomake data-driven decisions. The company’s artificial
intelligence technology analyzes hundreds of millions
of job postings and real-life career transitions toprovide insight into labor market patterns. This realtime strategic intelligence offers crucial insights, suchas what jobs are most in demand, the specific skills
employers need, and the career directions that offer thehighest potential for workers. For more information, visit
burning-glass.com.

Job Posting Data
To support these analyses, Burning Glass mined its
dataset of millions of job postings collected since 2010.
Burning Glass collects postings from over 45,000 online
job sites to develop a comprehensive, real-time portraitof labor market demand. It aggregates job postings,
removes duplicates, and extracts data from job postings
text. This includes information on job title, employer,industry, and region, as well as required experience,education, and skills.
Job postings are useful for understanding trends inthe labor market because they allow for a detailed,real-time look at the skills employers seek. To assess
the representativeness of job postings data, Burning
Glass conducts a number of analyses to compare the
distribution of job postings to the distribution of official
government and other third-party sources in the UnitedStates. The primary source of government data on U.S.
job postings is the Job Openings and Labor Turnover
Survey (JOLTS) program, conducted by the Bureau ofLabor Statistics.
To understand the share of job openings captured by
Burning Glass data, it is important to first note that
Burning Glass and JOLTS collect data on job postings
differently. Burning Glass data captures new postings:
A posting appears in the data only on the first month

APPENDIXTA B L E O F C O N T E N T S

204

Artificial IntelligenceIndex Report 2021

it is found and is considered a duplicate and removed
in subsequent months. JOLTS data captures active
postings: A posting appears in the data for every monththat it is still actively posted, meaning the same posting
can be counted in two or more consecutive months if it
has not been filled. To allow for apples-to-apples volume
comparison in postings, the Burning Glass data needs
to be inflated to account for active postings, not only
new postings. The number of postings from Burning
Glass can be inflated using the ratio of new jobs to active
jobs in Help Wanted OnLine™ (HWOL), a method usedin Carnevale, Jayasundera and Repnikov (2014). Basedon this calculation, the share of jobs online as captured
by Burning Glass is roughly 85% of the jobs captured in
JOLTS in 2016.
The labor market demand captured by Burning Glassdata represents over 85% of the total labor demand.
Jobs not posted online are usually in small businesses(the classic example being the “Help Wanted” sign in therestaurant window) and union hiring halls.

Measuring Demand for AI
In order to measure the demand by employers of AI skills,
Burning Glass uses its skills taxonomy of over 17,000
skills. The list of AI skills from Burning Glass data areshown below, with associated skill clusters. While someskills are considered to be in the AI cluster specifically,
for the purposes of this report, all skills below wereconsidered AI skills. A job posting was considered an AI
job if it requested one or more of these skills.
Artificial Intelligence: Expert System, IBM Watson,
IPSoft Amelia, Ithink, Virtual Agents, Autonomous
Systems, Lidar, OpenCV, Path Planning, Remote Sensing

APPENDIXC H A P T E R 3:
ECONOMY

Language Processing, Natural Language Toolkit (NLTK),
Nearest Neighbor Algorithm, OpenNLP, Sentiment
Analysis/Opinion Mining, Speech Recognition, Text
Mining, Text to Speech (TTS), Tokenization, Word2Vec
Neural Networks: Caffe Deep Learning Framework,
Convolutional Neural Network (CNN), Deep Learning,
Deeplearning4j, Keras, Long Short-Term Memory (LSTM),
MXNet, Neural Networks, Pybrain, Recurrent Neural
Network (RNN), TensorFlow
Machine Learning: AdaBoost algorithm, Boosting(Machine Learning), Chi Square Automatic Interaction
Detection (CHAID), Classification Algorithms, Clustering
Algorithms, Decision Trees, Dimensionality Reduction,
Google Cloud Machine Learning Platform, Gradientboosting, H2O (software), Libsvm, Machine Learning,
Madlib, Mahout, Microsoft Cognitive Toolkit, MLPACK(C++ library), Mlpy, Random Forests, Recommender
Systems, Scikit-learn, Semi-Supervised Learning,
Supervised Learning (Machine Learning), Support
Vector Machines (SVM), Semantic Driven Subtractive
Clustering Method (SDSCM), Torch (Machine Learning),
Unsupervised Learning, Vowpal, Xgboost
Robotics: Blue Prism, Electromechanical Systems,
Motion Planning, Motoman Robot Programming, Robot
Framework, Robotic Systems, Robot Operating System(ROS), Robot Programming, Servo Drives / Motors,
Simultaneous Localization and Mapping (SLAM)
Visual Image Recognition: Computer Vision, Image
Processing, Image Recognition, Machine Vision, Object
Recognition

Natural Language Processing (NLP): ANTLR, Automatic
Speech Recognition (ASR), Chatbot, Computational
Linguistics, Distinguo, Latent Dirichlet Allocation, Latent
Semantic Analysis, Lexalytics, Lexical Acquisition,
Lexical Semantics, Machine Translation (MT), Modular
Audio Recognition Framework (MARF), MoSes, Natural

APPENDIXTA B L E O F C O N T E N T S

205

Artificial IntelligenceIndex Report 2021

N E T BA S E Q U I D
Prepared by Julie Kim
NetBase Quid is a big data analytics platform thatinspires full-picture thinking by drawing connections
across massive amounts of unstructured data. The
software applies advanced natural language processing
technology, semantic analysis, and artificial intelligencealgorithms to reveal patterns in large, unstructured
datasets and to generate visualizations that allow users
to gain actionable insights. NetBase Quid uses Boolean
query to search for focus areas, topics, and keywords
within the archived news and blogs, companies, andpatents database, as well as any custom uploaded
datasets. This can filter out the search by publisheddate time frame, source regions, source categories,
or industry categories on the news—and by regions,investment amount, operating status, organization type(private/public), and founding year within the companies
database. NetBase Quid then visualizes these data points
based on the semantic similarity.

Search, Data Sources, and Scope
Here 3.6 million public and private company profiles
from multiple data sources are indexed in order tosearch across company descriptions, while filtering andincluding metadata ranging from investment information
to firmographic information, such as founded year, HQ
location, and more. Company information is updated
on a weekly basis. Quid algorithm reads a big amount oftext data from each document (news article, company
descriptions, etc.) to make links between different
documents based on their similar language. This process isrepeated at an immense scale, which produces a networkwith different clusters identifying distinct topics or focus
areas. Trends are identified based on keywords, phrases,
people, companies, institutions that Quid identifies, andthe other metadata that is put into the software.

Data
Organization data is embedded from CapIQ and
Crunchbase. These companies include all types of

APPENDIXTA B L E O F C O N T E N T S

APPENDIXC H A P T E R 3:
ECONOMYcompanies (private, public, operating, operating asa subsidiary, out of business) throughout the world.
The investment data includes private investments,
M&A, public offerings, minority stakes made by PE/VCs,
corporate venture arms, governments, and institutions
both within and outside the United States. Some data
is simply unreachable—for instance, when the investors
are undisclosed or the funding amounts by investors are
undisclosed. Quid also embeds firmographic informationsuch as founded year and HQ location.
NetBase Quid embeds CapIQ data as a default andadds in data from Crunchbase for the ones that are notcaptured in CapIQ. This yields not only comprehensive
and accurate data on all global organizations, but it also
captures early-stage startups and funding events data.
Company information is uploaded on a weekly basis.

Methodology
Boolean query is used to search for focus areas, topics,
and keywords within the archived company database,within their business descriptions and websites. We canfilter out the search results by HQ regions, investment
amount, operating status, organization type (private/public), and founding year. Quid then visualizes thesecompanies. If there are more than 7,000 companies fromthe search result, Quid selects the 7,000 most relevant
companies for visualization based on the language
algorithm.
Boolean Search: “artificial intelligence” or “AI” or“machine learning” or “deep learning”
Companies:• Chart 3.2.1: Global AI & ML companies that havebeen invested (private, IPO, M&A) from 01/01/2011 to
12/31/2020.
• Chart 3.2.2–3.2.6: Global AI & ML companies that haveinvested over USD 400,000 for the last 10 years (January
1, 2011 to December 31, 2020)—7,000 companies outof 7,500 companies have been selected through Quid’s
relevance algorithm.

206

Artificial IntelligenceIndex Report 2021

Target Event Definitions• Private investments: A private placement is a private
sale of newly issued securities (equity or debt) by acompany to a selected investor or a selected groupof investors. The stakes that buyers take in private
placements are often minority stakes (under 50%),although it is possible to take control of a company
through a private placement as well, in which casethe private placement would be a majority stake
investment.
• Minority investment: These refer to minority stakeacquisitions in Quid, which take place when the buyer
acquires less than 50% of the existing ownership stakein entities, asset product, and business divisions.
• M&A: This refers to a buyer acquiring more than 50% of
the existing ownership stake in entities, asset product,
and business divisions.

APPENDIXC H A P T E R 3:
ECONOMYI N T E R N AT I O N A L F E D E R AT I O N O F
R O B OT I C S

SourceData was received directly from the International
Federation of Robotics’ (IFR) 2020 World Robotics Report.
Learn more about IFR.

Methodology
The data displayed is the number of industrial robots
installed by country. Industrial robots are defined by theISO 8373:2012 standard. See more information on IFR’s
methodology.

NuanceM C K I N S E Y & C O M PA N Y• It is unclear how to identify what percentage of robot
units run software that would be classified as “AI,” and it
is unclear to what extent AI development contributes toindustrial robot usage.
• This metric was called “robot imports” in the 2017 AI
Index Report.

This survey was written, filled, and analyzed by McKinsey &Company. You can find additional results from the Global
AI Survey here.

Prepared by Jeffrey Banner and Steven Nichols

Methodology

SourceSOURCEThe survey was conducted online and was in the field from
June 9, 2020, to June 19, 2020, and garnered responses
from 2,395 participants representing the full range ofregions, industries, company sizes, functional specialties,
and tenures. Of those respondents, 1,151 said theirorganizations had adopted AI in at least one function andwere asked questions about their organizations’ AI use.
To adjust for differences in response rates, the data areweighted by the contribution of each respondent’s nation
to global GDP. McKinsey also conducted interviews withexecutives between May and August 2020 about theircompanies’ use of AI. All quotations from executives weregathered during those interviews.

P R AT T L E ( E A R N I N G CA L L S O N LY )Liquidnet provides sentiment data that predicts
the market impact of central bank and corporate
communications. Learn more about Liquidnet here.

Note
Survey respondents are limited by their perception of theirorganization’s AI adoption.

APPENDIXTA B L E O F C O N T E N T S

207

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 4:A I E D U CAT I O N

CHAPTER 4: AI EDUCATIONC R A TAU L B E E S U R V E Y
Prepared by Betsy Bizot (CRA senior research associate)and Stu Zweben (CRA survey chair, professor emeritus at
The Ohio State University)

SourceComputing Research Association (CRA) members
are 200-plus North American organizations active incomputing research: academic departments of computerscience and computer engineering; laboratories andcenters in industry, government, and academia; andaffiliated professional societies (AAAI, ACM, CACS/AIC,IEEE Computer Society, SIAM USENIX). CRA’s mission
is to enhance innovation by joining with industry,
government, and academia to strengthen research andadvanced education in computing. Learn more about
CRA here.

Methodology
CRA Taulbee Survey gathers survey data during the fallof each academic year by reaching out to over 200 PhDgranting departments. Details about the Taulbee Survey
can be found here. Taulbee does not directly survey thestudents. The department identifies each new PhD’s areaof specialization as well as their type of employment.
Data is collected from September to January of eachacademic year for PhDs awarded in the previous
academic year. Results are published in May after datacollection closes. So the 2019 data points were newly
available last spring, and the numbers provided for 2020
will be available in May 2021.

APPENDIXTA B L E O F C O N T E N T S

The CRA Taulbee Survey is sent only to doctoral
departments of computer science, computer engineering,and information science/systems. Historically, (a)Taulbee covers 1/4 to 1/3 of total BS CS recipients in
the United States; (b) the percent of women earning
bachelor’s degrees is lower in the Taulbee schools thanoverall; and (c) Taulbee tracks the trends in overall CS
production.

Nuances• Of particular interest in PhD job market trends are
the metrics on the AI PhD area of specialization. The
categorization of specialty areas changed in 2008 andwas clarified in 2016. From 2004-2007, AI and robotics
were grouped; from 2008-present, AI is separate; 2016clarified to respondents that AI includes ML.
• Notes about the trends in new tenure-track hires(overall and particularly at AAU schools): In the 2018
Taulbee Survey, for the first time, we asked how many
new hires had come from the following sources: new
PhD, postdoc, industry, and other academic. Results
indicated that 29% of new assistant professors camefrom another academic institution.
• Some may have been teaching or research faculty
rather than tenure-track, but there is probably somemovement between institutions, meaning the total
number hired overstates the total who are actually new.

208

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 4:A I E D U CAT I O N

A I I N D E X E D U CAT I O N S U R V E Y
Prepared by Daniel Zhang (Stanford Institute for HumanCentered Artificial Intelligence)

Methodology
The survey was distributed to 73 universities online overthree waves from November 2020 to January 2021 andcompleted by 18 universities, a 24.7% response rate. The
selection of universities is based on the World University
Rankings 2021 and Emerging Economies University
Rankings 2020 by The Times Higher Education.
The 18 universities are:
• Belgium: Katholieke Universiteit Leuven• Canada: McGill University• China: Shanghai Jiao Tong University, Tsinghua University• Germany: Ludwig Maximilian University of Munich,Technical University of Munich• Russia: Higher School of Economics, Moscow Institute ofPhysics and Technology• Switzerland: École Polytechnique Fédérale de Lausanne• United Kingdom: University of Cambridge• United States: California Institute of Technology,
Carnegie Mellon University (Department of Machine
Learning), Columbia University, Harvard University,
Stanford University, University of Wisconsin–Madison,
University of Texas at Austin, Yale University

Key Definitions• Major or a study program: a set of required and elective
courses in an area of discipline—such as AI—that leads toa bachelor’s degree upon successful completion.
• Course: a set of classes that require a minimum of 2.5
class hours (including lecture, lab, TA hours, etc.) per
week for at least 10 weeks in total. Multiple courses withthe same titles and numbers count as one course.

APPENDIXTA B L E O F C O N T E N T S• Practical Artificial Intelligence Models - Keywords:
Adaptive learning, AI Application, Anomaly detection,
Artificial general intelligence, Artificial intelligence,
Audio processing, Automated vehicle, Automatictranslation, Autonomous system, Autonomous vehicle,
Business intelligence, Chatbot, Computational creativity,
Computational linguistics, Computational neuroscience,
Computer vision, Control theory, Cyber physical steam,
Deep learning, Deep neural network, Expert system, Face
recognition, Human-AI interaction, Image processing,Image recognition, Inductive programming, Intelligencesoftware, Intelligent agent, Intelligent control, Intelligent
software development, Intelligence system, Knowledge
representation and reasoning, Machine learning, Machinetranslation, Multi-agent system, Narrow artificial
intelligence, Natural language generation, Natural
language processing, Natural language understanding,
Neural network, Pattern recognition, Predictive analysis,
Recommender system, Reinforcement learning, Robotsystem, Robotics, Semantic web, Sentiment analysis,
Service robot, Social robot, Sound synthesis, Speaker
identification, Speech processing, Speech recognition,
Speech synthesis, Strong artificial intelligence,Supervised learning, Support vector machine, Swarm
intelligence, Text mining, Transfer learning, Unsupervised
learning, Voice recognition, Weak artificial intelligence(Adapted from: Joint Research Centre, European
Commission, p.68)• AI Ethics - Keywords: Accountability, Consent,
Contestability, Ethics, Equality, Explainability, Fairness,
Non-discrimination, Privacy, Reliability, Safety, Security,
Transparency, Trustworthy ai, Uncertainty, Well-being(Adapted from: Joint Research Centre, European
Commission, p.68)

209

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 4:A I E D U CAT I O N

E U ACA D E M I C O F F E R I N G , J O I N T
R E S E A R C H C E N T E R, E U R O P E A NCOMMISSION
Prepared by Giuditta De-Prato, Montserrat López Cobo,and Riccardo Righi

SourceThe Joint Research Centre (JRC) is the European
Commission’s science and knowledge service. The JRC
employs scientists to carry out research in order to provide
independent scientific advice and support to EU policy.
Learn more about JRC here.

Methodology
By means of text-mining techniques, the study identifies
AI-related education programs from the programs’descriptions present in JRC’s database. To query the
database, a list of domain-specific keywords is obtained
through a multistep methodology involving (i) selectionof top keywords from AI-specific scientific journals;(ii) extraction of representative terms of the industrial
dimension of the technology; (iii) topic modeling; and (iv)validation by experts. In this edition, the list of keywords
has been enlarged to better cover certain AI subdomains
and to expand to related transversal domains, suchas philosophy and ethics in AI. Then the keywords aregrouped into categories, which are used to analyze thecontent areas taught in the identified programs. The
content areas used are adapted from the JRC report“Defining Artificial Intelligence: Towards an Operational
Definition and Taxonomy of Artificial Intelligence,”
conducted in the context of AI Watch.
The education programs are classified into specialized
and broad, according to the depth with which they
address artificial intelligence. Specialized programs

APPENDIXTA B L E O F C O N T E N T S

are those with a strong focus in AI, e.g., “automationand computer vision” or “advanced computer science(computational intelligence).” Broad programs target theaddressed domain, but in a more generic way, usuallyaiming at building wider profiles or making reference tothe domain in the framework of a program specialized ina different discipline (e.g., biomedical engineering).
Due to some methodological improvements introducedin this edition, namely the addition of new keywords, astrict comparison is not possible. Still, more than 90%
of all detected programs in this edition are triggered by
keywords present in the 2019 study.
The original source on which queries are performed is the
Studyportals’ database, which is made up of over 207,000
programs from 3,700 universities in over 120 countries.
Studyportals collects information from institutions’
websites, and their database is regularly updated. This
source, although offering the widest coverage among
all those identified, still suffers from some lack ofcoverage, mostly because it only tracks English-language
programs. This poses a comparability issue between
English-native-speaking countries and the rest, but also
between countries with differing levels of incorporationof English as a teaching language in higher education.
Bachelor’s-level studies are expected to be more
affected by this fact, where the offer is mostly taught ina native language, unlike master’s, which attracts moreinternational audiences and faculties. As a consequence,
this study may be showing a partial picture of the level ofinclusion of advanced digital skills in bachelor’s degree
programs.

210

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 5 : E T H I CA L
CHALLENGES OFA I A P P L I CAT I O N S

CHAPTER 5: ETHICAL CHALLENGES
OF AI APPLICATIONSN E T BA S E Q U I D
Prepared by Julie Kim
Quid is a data analytics platform within the NetBase
Quid portfolio that applies advanced natural language
processing technology, semantic analysis, and artificial
intelligence algorithms to reveal patterns in large,
unstructured datasets and generate visualizations to allow
users to gain actionable insights. Quid uses Boolean query
to search for focus areas, topics, and keywords within
the archived news and blogs, companies, and patents
database, as well as any custom uploaded datasets. Users
can then filter their search by published date time frame,
source regions, source categories, or industry categories
on the news; and by regions, investment amount,
operating status, organization type (private/public), andfounding year within the companies’ database. Quidthen visualizes these data points based on the semantic
similarity.

Network
Searched for [AI technology keywords + Harvard ethics
principles keywords] global news from January 1, 2020, toDecember 31, 2020.
Search Query: (AI OR [“artificial intelligence”](“artificial
intelligence” OR “pattern recognition” OR algorithms)
OR [“machine learning”](“machine learning” OR“predictive analytics” OR “big data” OR “patternrecognition” OR “deep learning”) OR [“natural language”](“natural language” OR “speech recognition”) OR NLP

APPENDIXTA B L E O F C O N T E N T S

OR “computer vision” OR [“robotics”](“robotics” OR“factory automation”) OR “intelligent systems” OR [“facialrecognition”](“facial recognition” OR “face recognition”
OR “voice recognition” OR “iris recognition”) OR[“image recognition”](“image recognition” OR “patternrecognition” OR “gesture recognition” OR “augmented
reality”) OR [“semantic search”](“semantic search”
OR “data-mining” OR “full-text search” OR “predictive
coding”) OR “semantic web” OR “text analytics” OR“virtual assistant” OR “visual search”) AND (ethics OR“human rights” OR “human values” OR “responsibility” OR“human control” OR “fairness” OR discrimination OR nondiscrimination OR “transparency” OR “explainability” OR“safety and security” OR “accountability” OR “privacy” )News Dataset Data Source
Quid indexes millions of global-source, English-language
news articles and blog posts from LexisNexis. The platform
has archived news and blogs from August 2013 to thepresent, updating every 15 minutes. Sources include over
60,000 news sources and over 500,000 blogs.

Visualization in Quid Software
Quid uses Boolean query to search for topics, trends, andkeywords within the archived news database, with theability to filter results by the published date time frame,
source regions, source categories, or industry categories.
(In this case, we only looked at global news published fromJanuary 1, 2020, to December 31, 2020.) Quid then selects
the 10,000 most relevant stories using its NLP algorithmand visualizes de-duplicated unique articles.

211

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 5 : E T H I CA L
CHALLENGES OFA I A P P L I CAT I O N SETHICS IN AI CONFERENCES
Prepared by Marcelo Prates, Pedro Avelar, and Luis C.
Lamb

SourcePrates, Marcelo, Pedro Avelar, Luis C. Lamb. 2018.
On Quantifying and Understanding the Role of Ethics in
AI Research: A Historical Account of Flagship Conferences
and Journals. September 21, 2018.

Methodology
The percent of keywords has a straightforward
interpretation: For each category (classical/trending/
ethics), the number of papers for which the title (orabstract, in the case of the AAAI and NeurIPS figures)contains at least one keyword match. The percentages do
not necessarily add up to 100% (e.g, classical/trending/
ethics are not mutually exclusive). One can have a paperwith matches on all three categories.
To achieve a measure of how much Ethics in AI is
discussed, ethics-related terms are searched for in thetitles of papers in flagship AI, machine learning, androbotics conferences and journals.
The ethics keywords used were the following:
Accountability, Accountable, Employment, Ethic, Ethical,
Ethics, Fool, Fooled, Fooling, Humane, Humanity, Law,
Machine Bias, Moral, Morality, Privacy, Racism, Racist,
Responsibility, Rights, Secure, Security, Sentience,
Sentient, Society, Sustainability, Unemployment, andWorkforce.
The classical and trending keyword sets were compiled
from the areas in the most cited book on AI by Russell
and Norvig [2012] and from curating terms from the
keywords that appeared most frequently in paper titles
over time in the venues.

APPENDIXTA B L E O F C O N T E N T S

The keywords chosen for the classical keywords category
were:
Cognition, Cognitive, Constraint Satisfaction, Game
Theoretic, Game Theory, Heuristic Search, Knowledge
Representation, Learning, Logic, Logical, Multiagent,
Natural Language, Optimization, Perception, Planning,
Problem Solving, Reasoning, Robot, Robotics, Robots,
Scheduling, Uncertainty, and Vision.
The curated trending keywords were:
Autonomous, Boltzmann Machine, Convolutional
Networks, Deep Learning, Deep Networks, Long ShortTerm Memory, Machine Learning, Mapping, Navigation,
Neural, Neural Network, Reinforcement Learning,
Representation Learning, Robotics, Self Driving, SelfDriving, Sensing, Slam, Supervised/Unsupervised
Learning, and Unmanned.
The terms searched for were based on the issues exposed
and identified in papers below, and also on the topics
called for discussion in the First AAAI/ACM Conference on
AI, Ethics, and Society.
J. Bossmann. Top 9 Ethical Issues in Artificial Intelligence.
2016. World Economic Forum.
Emanuelle Burton, Judy Goldsmith, Sven Koenig,
Benjamin Kuipers, Nicholas Mattei, and Toby Walsh.
Ethical Considerations in Artificial Intelligence Courses. AI
Magazine, 38(2):22–34, 2017.
The Royal Society Working Group, P. Donnelly, R.
Browsword, Z. Gharamani, N. Griffiths, D. Hassabis, S.
Hauert, H. Hauser, N. Jennings, N. Lawrence, S. Olhede,
M. du Sautoy, Y.W. Teh, J. Thornton, C. Craig, N. McCarthy,
J. Montgomery, T. Hughes, F. Fourniol, S. Odell, W. Kay,
T. McBride, N. Green, B. Gordon, A. Berditchevskaia, A.
Dearman, C. Dyer, F. McLaughlin, M. Lynch, G. Richardson,
C. Williams, and T. Simpson. Machine Learning: The
Power and Promise of Computers That Learn by Example.
The Royal Society, 2017.

212

Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 5 : E T H I CA L
CHALLENGES OFA I A P P L I CAT I O N S

Conference and Public Venue - Sample
The AI group contains papers from the main artificial
intelligence and machine learning conferences such
as AAAI, IJCAI, ICML, and NIPS and also from both theArtificial Intelligence Journal and the Journal of Artificial
Intelligence Research (JAIR).
The robotics group contains papers published in the IEEE
Transactions on Robotics and Automation (now known asIEEE Transactions on Robotics), ICRA, and IROS.
The CS group contains papers published in the mainstream
computer science venues such as the Communications of
the ACM, IEEE Computer, ACM Computing Surveys, and the
ACM and IEEE Transactions.

Codebase
The code and data are hosted in this GitHub repository.

APPENDIXTA B L E O F C O N T E N T S

213

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 6:
DIVERSITY IN AI

CHAPTER 6: DIVERSITY IN AI
LINKEDINAI Skills Penetration
The aim of this indicator is to measure the intensity ofAI skills in an entity (in a particular country, industry,
gender, etc.) through the following methodology:• Compute frequencies for all self-added skills by
LinkedIn members in a given entity (occupation,industry, etc.) in 2015–2020.
• Re-weight skill frequencies using a TF-IDF model to get
the top 50 most representative skills in that entity. These
50 skills compose the “skill genome” of that entity.
• Compute the share of skills that belong to the AI skill
group out of the top skills in the selected entity.
Interpretation: The AI skill penetration rate signals theprevalence of AI skills across occupations, or the intensitywith which LinkedIn members utilize AI skills in theirjobs. For example, the top 50 skills for the occupation ofengineer are calculated based on the weighted frequency
with which they appear in LinkedIn members’ profiles. If
four of the skills that engineers possess belong to the AI
skill group, this measure indicates that the penetration of
AI skills is estimated to be 8% among engineers (e.g., 4/50).

Global Comparison: By Gender
The relative AI skill penetration by country for genderprovides an in-depth decomposition of AI skills
penetration across female and male labor pools andsample countries.
Interpretation: A country’s relative AI skill penetration
rate of 2 for women means that the average penetrationof AI skills among women in that country is two times
the global average across the same set of occupations
among women. If, in the same country, the relative AI
skill penetration rate for men is 1.9, this indicates thatthe average penetration of AI skills among women inthat country is 5% higher than that of men (calculated by
dividing 1.9 by 2 and then subtracting 1, or 2/1.9-1) for
the same set of occupations.

Relative AI Skills Penetration
To allow for skills penetration comparisons acrosscountries, the skills genomes are calculated and a
relevant benchmark is selected (e.g., global average).
A ratio is then constructed between a country’s andthe benchmark’s AI skills penetrations, controlling foroccupations.
Interpretation: A country’s relative AI skills penetrationof 1.5 indicates that AI skills are 1.5 times as frequent as
in the benchmark, for an overlapping set of occupations.

APPENDIXTA B L E O F C O N T E N T S

214Artificial IntelligenceIndex Report 2021

APPENDIXC H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

CHAPTER 7: AI POLICY AND
NATIONAL STRATEGIESB LO O M B E R G G OV E R N M E N T
Bloomberg Government (BGOV) is a subscriptionbased market intelligence service designed to make
U.S. government budget and contracting data moreaccessible to business development and government
affairs professionals. BGOV’s proprietary tools ingest
and organize semi-structured government data sets
and documents, enabling users to track and forecast
investment in key markets.

Methodology
The BGOV data included in this section was drawn fromthree original sources:
Contract Spending: BGOV’s Contracts Intelligence Tool
ingests on a twice-daily basis all contract spending data
published to the beta.SAM.gov Data Bank, and structures
the data to ensure a consistent picture of government
spending over time. For the section “U.S. Government
Contract Spending,” BGOV analysts used FPDS-NG data,
organized by the Contracts Intelligence Tool, to build amodel of government spending on artificial intelligencerelated contracts in the fiscal years 2000 through 2021.
BGOV’s model used a combination of governmentdefined produce service codes and more than 100
AI-related keywords and acronyms to identify AI-related
contract spending.
Defense RDT&E Budget: BGOV organized all 7,057
budget line items included in the RDT&E budget request
based on data available on the DOD Comptroller website.
For the section “U.S. Department of Defense (DOD)
Budget,” BGOV used a set of more than a dozen AIspecific keywords to identify 305 unique budget activities
related to artificial intelligence and machine learning
worth a combined USD 5.0 billion in FY 2021.
Congressional Record (available on Congressional
Record website): BGOV maintains a repository of

APPENDIXTA B L E O F C O N T E N T S

congressional documents, including bills, amendments,
bill summaries, Congressional Budget Officeassessments, reports published by congressional
committees, Congressional Research Service (CRS), andothers. For the section “U.S. Congressional Record,”
BGOV analysts identified all legislation (passed or
introduced), congressional committee reports, and
CRS reports that referenced one or more of a dozen AIspecific keywords. Results are organized by a two-year
congressional session.

LIQUIDNET
Prepared by Jeffrey Banner and Steven Nichols

SourceLiquidnet provides sentiment data that predicts
the market impact of central bank and corporate
communications. Learn more about Liquidnet here.

Examples of Central Bank Mentions
Here are some examples of how AI is mentioned bycentral banks: In the first case, China uses a geopoliticalenvironment simulation and prediction platformthat works by crunching huge amounts of data andthen providing foreign policy suggestions to Chinese
diplomats or the Bank of Japan use of AI prediction
models for foreign exchange rates. For the second
case, many central banks are leading communications
through either official documents—for example, on
July 25, 2019, the Dutch Central Bank (DNB) published
Guidelines for the use of AI in financial services andlaunched its six “SAFEST” principles for regulated firms
to use AI responsibly—or a speech on June 4, 2019, by
the Bank of England’s Executive Director of U.K. Deposit
Takers Supervision James Proudman, titled “Managing
Machines: The Governance of Artificial Intelligence,”
focused on the increasingly important strategic issue of
how boards of regulated financial services should use AI.

215

Artificial IntelligenceIndex Report 2021

M C K I N S E Y G LO BA L I N S T I T U T E

SourceData collection and analysis was performed by the
McKinsey Global Institute (MGI).

Canada (House of Commons)
Data was collected using the Hansard search feature on
Parliament of Canada website. MGI searched for the terms“Artificial Intelligence” and “Machine Learning” (quotes
included) and downloaded the results as a CSV. The date
range was set to “all debates.” Data is as of Dec. 31, 2020.
Data are available online from Aug. 31, 2002.
Each count indicates that Artificial Intelligence or Machine
Learning was mentioned in a particular comment or remark
during the proceedings of the House of Commons. Thismeans that within an event or conversation, if a member
mentions AI or ML multiple times within their remarks, itwill appear only once. However if, during the same event,
the speaker mentions AI or ML in separate comments (withother speakers in between), it will appear multiple times.
Counts for Artificial Intelligence or Machine Learning are
separate, as they were conducted in separate searches.
Mentions of the abbreviations AI or ML are not included.

United Kingdom (House of Commons, House ofLords, Westminster Hall, and Committees)
Data was collected using the Find References feature of the
Hansard website of the U.K. Parliament. MGI searched forthe terms “Artificial Intelligence” and “Machine Learning”(quotes included) and catalogued the results. Data is asof Dec. 31, 2020. Data are available online from January 1,
1800 onward. Contains Parliamentary information licensed
under the Open Parliament Licence v3.0.
As in Canada, each count indicates that Artificial
Intelligence or Machine Learning was mentioned in aparticular comment or remark during a proceeding.
Therefore, if a member mentions AI or ML multiple times
within their remarks, it will appear only once. However
if, during the same event, the same speaker mentions
AI or ML in separate comments (with other speakers in
between), it will appear multiple times. Counts for Artificial

APPENDIXTA B L E O F C O N T E N T S

APPENDIXC H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E SIntelligence or Machine Learning are separate, as they
were conducted in separate searches. Mentions of theabbreviations AI or ML are not included.

United States (Senate and House ofRepresentatives)
Data was collected using the advanced search featureof the U.S. Congressional Record website. MGI searched
the terms “Artificial Intelligence” and “Machine Learning”(quotes included) and downloaded the results as a
CSV. The “word variant” option was not selected, andproceedings included Senate, House of Representatives,and Extensions of Remarks, but did not include the Daily
Digest. Data is as of Dec. 31, 2020, and data is available
online from the 104th Congress onward (1995).
Each count indicates that Artificial Intelligence or Machine
Learning was mentioned during a particular event
contained in the Congressional Record, including thereading of a bill. If a speaker mentioned AI or ML multiple
times within remarks, or multiple speakers mentioned AI or
ML within the same event, it would appear only once as aresult. Counts for Artificial Intelligence or Machine Learning
are separate, as they were conducted in separate searches.
Mentions of the abbreviations AI or ML are not included.

U.S. A I P O L I CY PA P E R

SourceData collection and analysis was performed by Stanford
Institute of Human-Centered Artificial Intelligence and AI Index.

Organizations
To develop a more nuanced understanding of thethought leadership that motivates AI policy, we tracked
policy papers published by 36 organizations across threebroad categories including:Think Tanks, Policy Institutes & Academia: This includes
organizations where experts (often from academia andthe political sphere) provide information and advice
on specific policy problems. We included the following
27 organizations: AI PULSE at UCLA Law, American
Enterprise Institute, Aspen Institute, Atlantic Council,
Berkeley Center for Long-Term Cybersecurity, Brookings216

Artificial IntelligenceIndex Report 2021

Institution, Carnegie Endowment for International Peace,
Cato Institute, Center for a New American Security,
Center for Strategic and International Studies, Council
on Foreign Relations, Georgetown Center for Security
and Emerging Technology (CSET), Harvard Belfer Center,
Harvard Berkman Klein Center, Heritage Foundation,
Hudson Institute, MacroPolo, MIT Internet Policy Research
Initiative, New America Foundation, NYU AI Now Institute,Princeton School of Public and International Affairs, RAND
Corporation, Rockefeller Foundation, Stanford Institutefor Human-Centered Artificial Intelligence (HAI), Stimson
Center, Urban Institute, Wilson Center.
Civil Society, Associations & Consortiums: Not-for profit
institutions including community-based organizations
and NGOs advocating for a range of societal issues. We
included the following nine organizations: Algorithmic
Justice League, Alliance for Artificial Intelligence in
Healthcare, Amnesty International, EFF, Future of Privacy
Forum, Human Rights Watch, IJIS, Institute for Electricaland Electronics Engineers, Partnership on AI
Industry & Consultancy: Professional practices providing
expert advice to clients and large industry players. We
included six prominent organizations in this space: Accenture,Bain & Co., BCG, Deloitte, Google AI, McKinsey & Company

Methodology
Each broad topic area is based on a collection of underlying
keywords that describes the content of the specific paper.
We included 17 topics that represented the majority of
discourse related to AI between 2019-2020. These topic
areas and the associated keywords are listed below.
• Health & Biological Sciences: medicine, healthcare
systems, drug discovery, care, biomedical research,
insurance, health behaviors, COVID-19, global health• Physical Sciences: chemistry, physics, astronomy, earthscience• Energy & Environment: Energy costs, climate change,energy markets, pollution, conservation, oil & gas,alternative energy• International Affairs & International Security:international relations, international trade, developing
countries, humanitarian assistance, warfare, regional

APPENDIXTA B L E O F C O N T E N T S

APPENDIXC H A P T E R 7:
A I P O L I CY A N D
N AT I O N A L S T R AT E G I E S

security, national security, autonomous weapons• Justice & Law Enforcement: civil justice, criminal justice,social justice, police, public safety, courts• Communications & Media: social media, disinformation,
media markets, deepfakes• Government & Public Administration: federal
government, state government, local government, public
sector efficiency, public sector effectiveness, government
services, government benefits, government programs,
public works, public transportation• Democracy: elections, rights, freedoms, liberties,
personal freedoms• Industry & Regulation: economy, antitrust, M&A,
competition, finance, management, supply chain,telecom, economic regulation, technical standards,
autonomous vehicle industry & regulation• Innovation & Technology: advancements andimprovements in AI technology, R&D, intellectual
property, patents, entrepreneurship, innovation
ecosystems, startups, computer science, engineering• Education & Skills: early childhood, K-12, higher
education, STEM, schools, classrooms, reskilling• Workforce & Labor: labor supply and demand, talent,
immigration, migration, personnel economics, future ofwork• Social & Behavioral Sciences: sociology, linguistics,
anthropology, ethnic studies, demography, geography,
psychology, cognitive science• Humanities: arts, music, literature, language,
performance, theater, classics, history, philosophy,
religion, cultural studies• Equity & Inclusion: biases, discrimination, gender,
race, socioeconomic inequality, disabilities, vulnerable
populations• Privacy, Safety & Security: anonymity, GDPR,
consumer protection, physical safety, human control,cybersecurity, encryption, hacking• Ethics: transparency, accountability, humanvalues, human rights, sustainability, explainability,interpretability, decision-making norms

217

Artificial IntelligenceIndex Report 2021

APPENDIX

G LO BA L A I
V I B R A N CYGLOBAL AI VIBRANCYOV E R V I E W
The tables below show the high-level pillar, sub-pillars, and indicators covered by the Global AI Vibrancy Tool. Eachsub-pillar is composed of individual indicators reported in the Global AI Vibrancy Codebook. There are 22 metrics intotal, with 14 metrics under Research and Development (R&D) pillar, 6 metrics under the Economy pillar, and 2 metrics
available under the Inclusion pillar specific to gender diversity. To aid data-driven decision-making to design national
policy strategies, the Global AI Vibrancy is available as a web tool.

R&D
SUB-PILLAR

VARIABLE

Conference Publications

Number of AI conference papers*

Conference Publications

Number of AI conference papers per capita

Conference Publications

Number of AI conference citations*

Conference Publications

Number of AI conference citations per capita

Journal Publications

Number of AI journal papers*

Journal Publications

Number of AI journal papers per capita

Journal Publications

Number of AI journal citations*

Journal Publications

Number of AI journal citations per capita

Innovation > Patents

Number of AI patents*

Innovation > Patents

Number of AI patents per capita

Innovation > Patents

Number of AI patent citations*Innovation > Patents

Number of AI patent citations per capita

Journal Publications > Deep Learning

Number of Deep Learning papers*

Journal Publications > Deep Learning

Number of Deep Learning papers per capita

ECONOMYINCLUSION

SUB-PILLAR

VARIABLE

SkillsRelative Skill Penetration

Labor

AI hiring index

Investment

Total AI Private Investment*

Investment

AI Private Investment per capita

Investment

Number of Startups Funded*

Investment

Number of funded startups per capita

APPENDIXTA B L E O F C O N T E N T S

SUB-PILLAR

VARIABLE

Gender Diversity

AI Skill Penetration (female)

Gender Diversity

Number of unique AI occupations(job titles), female

218

Artificial IntelligenceIndex Report 2021

The webtool allows users to adjust weights to each
metric based on their individual preference. The default
settings of the tool allow the user to select between threeweighting options:

All weights to midpoint
This button assigns equal weights to all indicators.

Only absolute metrics
This button assigns maximum weights to absolute
metrics. Per capita metrics are not considered.

Only per capita metrics
This button assigns maximum weights to per capita
metrics. Absolute metrics are not considered.
The user can adjust the weights to each metric based on
their preference. The charts automatically update when
any weight is changed.
The user can select “Global” or “National” view tovisualize the results. The “Global” view offers a crosscountry comparative view based on the weights selectedby the user. The “National” view offers a country deep
dive to assess which AI indicators a given country is
relatively better at. The country-metric specific values
are scaled (0-100), where 100 indicates that a given
country has the highest number in the global distribution
for that metric, and conversely small numbers like 0 or 1
indicates relatively low values in the global distribution
This can help identify areas for improvement andidentify national policy strategies to support a vibrant AI
ecosystem.

APPENDIX

G LO BA L A I
V I B R A N CYCONSTRU CTION OF THE GLO BA L A I
V IBRANCY: COM POSITE M E ASU R E

SourceThe data is collected by AI Index using diverse datasetsthat are referenced in the 2020 AI Index Report chapters.

Methodology
Step 1: Obtain, harmonize, and integrate data on
individual attributes across countries and time.
Step 2: Use Min-Max Scalar to normalize each country-year
specific indicator between 0-100.
Step 3: Take arithmetic Mean per country-indicator for agiven year.
Step 4: Build modular weighted for available pillars andindividual indicators.

Aggregate Measure
The AI Vibrancy Composite Index can be expressed in thefollowing equation:

where c represents a country and t represents year,isthe scaled (0-100) individual indicator,
is the weightassigned to individual indicators,
is weight specific
to one of the three high-level pillars and N is the number ofindicators available for a given country for a specific year.

Normalization
To adjust for differences in units of measurement andranges of variation, all 22 variables were normalized into
the [0, 100] range, with higher scores representing better
outcomes. A minimum-maximum normalization method
was adopted, given the minimum and maximum values ofeach variable respectively. Higher values indicate better
outcomes. The normalization formula is:

Coverage and Nuances
A threshold of 73% coverage was chosen to select the final listof countries based on an average of available data between
2015-2020. Russia and South Korea were added manually
due to their growing importance in the global AI landscape,even though they did not pass the 73% threshold.

APPENDIXTA B L E O F C O N T E N T S

219

Artificial IntelligenceIndex Report 2021

APPENDIX

G LO BA L A I
V I B R A N CY

R E S E A R C H A N D D E V E LO P M E N T I N D I CATO R SIDPILLAR

SUB-PILLAR

NAME

DEFINITION

SOURCE

1

Research andDevelopment

Conference
Publications

Number of AI
conference
papers*Total count of published AI conference papers
attributed to institutions in the given country.

Microsoft Academic
Graph (MAG)2

Research andDevelopment

Conference
Publications

Number of AI
conference
papers per capita

Total count of published AI conference papers
Microsoft Academic
attributed to institutions in the given country in percapita terms. The denominator is the population (in tens Graph (MAG)of millions) for a given year to obtain scaled values.

3

Research andDevelopment

Conference
Publications

Number of AI
conference
citations*

Total count of AI conference citations attributed to
institutions in the given country.

4

Research andDevelopment

Conference
Publications

Number of AI
conference
citations per
capitaTotal count of AI conference citations attributed to
institutions in the given country in per capita terms. The Microsoft Academic
denominator is the population (in tens of millions) for a Graph (MAG)given year to obtain scaled values.

5

Research andDevelopment

JournalPublications

Number of AI
journal papers*

Total count of published AI journal papers attributed to
institutions in the given country.

6

Research andDevelopment

JournalPublications

Number of AI
journal papers
per capita

Total count of published AI journal papers attributed to
institutions in the given country in per capita terms. The Microsoft Academic
denominator is the population (in tens of millions) for a Graph (MAG)given year to obtain scaled values.

7

Research andDevelopment

JournalPublications

Number of AI
journal citations*

Total count of AI journal citations attributed to
institutions in the given country.

8

Research andDevelopment

JournalPublications

Number of AI
journal citations
per capita

Total count of AI journal citations attributed to
institutions in the given country in per capita terms. The Microsoft Academic
denominator is the population (in tens of millions) for a Graph (MAG)given year to obtain scaled values.

9

Research andDevelopment

Innovation >Patents

Number of AI
patents*

Total count of published AI patents attributed to
institutions in the given country.

10

Research andDevelopment

Innovation >Patents

Total count of published AI patents attributed to
institutions in the given country in per capita terms. The Microsoft Academic
Number of AI
patents per capita denominator is the population (in tens of millions) for a Graph (MAG)given year to obtain scaled values.

11

Research andDevelopment

Innovation >Patents

Number of AI
patent citations*

Total count of published AI patents citations attributed
to institutions of originating patent filing.

Microsoft Academic
Graph (MAG)12

Research andDevelopment

Innovation >Patents

Number of AI
patent citations
per capita

Total count of published AI patent citations attributed
to institutions in the given country of originating patentfiling, in per capita terms. The denominator is the
population (in tens of millions) for a given year to obtainscaled values.

Microsoft Academic
Graph (MAG)13

Research andDevelopment

JournalPublications >Deep Learning

Number of deeplearning papers*

Total count of arXiv papers on Deep Learning attributed
arXiv, NESTA
to institutions in the given country.

14

Research andDevelopment

JournalPublications >Deep Learning

Number of deeplearning papers
per capita

Total count of arXiv papers on Deep Learning attributed
to institutions in the given country in per capita terms.
arXiv, NESTA
The denominator is the population (in tens of millions)for a given year to obtain scaled values.

APPENDIXTA B L E O F C O N T E N T S

Microsoft Academic
Graph (MAG)Microsoft Academic
Graph (MAG)Microsoft Academic
Graph (MAG)Microsoft Academic
Graph (MAG)220

Artificial IntelligenceIndex Report 2021

APPENDIX

G LO BA L A I
V I B R A N CY

E C O N O M Y I N D I CATO R SID15PILLAR

Economy

SUB-PILLAR

SkillsNAME

DEFINITION

SOURCE

Relative skill
penetration

Relative skill penetration rate measure is based ona method to compare how prevalent AI skills are at
the average occupation in each country against abenchmark (e.g. the global average), controlling for thesame set of occupations.

LinkedIn Economic
Graph

LinkedIn Economic
Graph

16

EconomyLabor

AI hiring index

AI hiring rate is the percentage of LinkedIn members
who had any AI skills (see the Appendix for the AI skill
grouping) on their profile and added a new employer
to their profile in the same month the new job began,divided by the total number of LinkedIn members in the
country. This rate is then indexed to the average monthin 2015-2016; for example, an index of 1.05 indicates ahiring rate that is 5% higher than the average month in2015-2016.

17

EconomyInvestment

Total Amount ofFunding*

Total amount of private investment funding received for Crunchbase, CapIQ,
AI startups (nominal USD).
NetBase Quid

18

EconomyInvestment

Total per capita
funding

Total amount of private investment funding received for
AI startups in per capita terms. The denominator is the Crunchbase, CapIQ,
population (in tens of millions) for a given year to obtain NetBase Quidappropriately scaled values.

19

EconomyInvestment

Number ofcompanies
funded*Total number of AI companies founded in the given
country.

Crunchbase, CapIQ,
NetBase Quid

20

EconomyInvestment

Number ofcompanies
funded per capita

Total number of AI companies founded in the given
country in per capita terms. The denominator is the
population (in tens of millions) for a given year to obtainappropriately scaled values.

Crunchbase, CapIQ,
NetBase Quid

I N C LU S I O N I N D I CATO R SIDPILLAR

SUB-PILLAR

NAME

DEFINITION

SOURCE

Relative skill penetration rate measure is based ona method to compare how prevalent AI skills are at
the average occupation in each country against abenchmark (e.g. the global average), controlling for thesame set of occupations.

LinkedIn Economic
Graph

LinkedIn Economic
Graph

21

Inclusion

GenderDiversity

AI skill
penetration(female)22

Inclusion

GenderDiversity

Number of unique
Number of unique AI occupations (or job titles) with
AI occupations
high AI skill penetration for females in a given country.
(job titles), female

APPENDIXTA B L E O F C O N T E N T S

221

Artificial
IntelligenceIndex Report 2021