Artificial IntelligenceIndex Report 2021

CHAPTER 1:

Research &Development

CHAPTER 1 PREVIEW

Artificial IntelligenceIndex Report 2021

1

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

CHAPTER 1:

Chapter Preview
Overview

3

Chapter Highlights

4

1.1 PUBLICATIONS

5

Peer-Reviewed AI Publications

AI Patents

18

Overview

18

arXiv Publications

19

Overview

19

5

By Region

19

Overview

5

By Geographic Area20

By Region

5

By Field of Study21

By Geographic Area

7

By Institutional Affiliation

8

Academic-Corporate Collaboration

10

1.2 CONFERENCES

23

12

Conference Attendance

23

Overview

12

By Region

13

Highlight: Corporate Representation
at AI Research Conferences

25

By Geographic Area

14

Citation

14

1.3 AI OPEN-SOURCE SOFTWARELIBRARIES

26

GitHub Stars

26

APPENDIX

28

AI Journal Publications

AI Conference Publications

15

Overview

15

By Region

16

By Geographic Area

17

Citation

17

Highlight: Deep Learning Papers on arXiv 22

ACCESS THE PUBLIC DATACHAPTER 1 PREVIEW

2

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OV E R V I E W

Overview
The report opens with an overview of the research and development (R&D)
efforts in artificial intelligence (AI) because R&D is fundamental to AI
progress. Since the technology first captured the imagination of computerscientists and mathematicians in the 1950s, AI has grown into a major
research discipline with significant commercial applications. The numberof AI publications has increased dramatically in the past 20 years. The riseof AI conferences and preprint archives has expanded the dissemination ofresearch and scholarly communications. Major powers, including China, theEuropean Union, and the United States, are racing to invest in AI research.
The R&D chapter aims to capture the progress in this increasingly complex
and competitive field.
This chapter begins by examining AI publications—from peer-reviewed
journal articles to conference papers and patents, including the citation
impact of each, using data from the Elsevier/Scopus and Microsoft
Academic Graph (MAG) databases, as well as data from the arXiv paper
preprint repository and Nesta. It examines contributions to AI R&Dfrom major AI entities and geographic regions and considers how thosecontributions are shaping the field. The second and third sections discuss
R&D activities at major AI conferences and on GitHub.

CHAPTER 1 PREVIEW

3

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

CHAPTERHIGHLIGHTS

CHAPTER HIGHLIGHTS• The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher
percentage growth than from 2018 to 2019 (19.6%).
• In every major country and region, the highest proportion of peer-reviewed AI papers
comes from academic institutions. But the second most important originators aredifferent: In the United States, corporate-affiliated research represents 19.2% of the total
publications, whereas government is the second most important in China (15.6%) and theEuropean Union (17.2%).
• In 2020, and for the first time, China surpassed the United States in the share of AI
journal citations in the world, having briefly overtaken the United States in the overall
number of AI journal publications in 2004 and then retaken the lead in 2017. However,the United States has consistently (and significantly) more cited AI conference papers
than China over the last decade.
• In response to COVID-19, most major AI conferences took place virtually and registereda significant increase in attendance as a result. The number of attendees across nine
conferences almost doubled in 2020.
• In just the last six years, the number of AI-related publications on arXiv grew by morethan sixfold, from 5,478 in 2015 to 34,736 in 2020.
• AI publications represented 3.8% of all peer-reviewed scientific publications worldwide
in 2019, up from 1.3% in 2011.

CHAPTER 1 PREVIEW

4

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

AI publications include peer-reviewed publications, journal articles, conference papers, and patents. To track trends among thesepublications and to assess the state of AI R&D activities around the world, the following datasets were used: the Elsevier/Scopus
database for peer-reviewed publications; the Microsoft Academic Graph (MAG) database for all journals, conference papers, andpatent publications; and arXiv and Nesta data for electronic preprints.

1.1 PUBLICATIONSP E E R - R E V I E W E D A I P U B L I CAT I O N SThis section presents data from the Scopus database by
Elsevier. Scopus contains 70 million peer-reviewed research
items curated from more than 5,000 international publishers.
The 2019 version of the data shown below is derived froman entirely new set of publications, so figures of all peerreviewed AI publications differ from those in previous years’
AI Index reports. Due to changes in the methodology forindexing publications, the accuracy of the dataset increased
from 80% to 84% (see the Appendix for more details).

Overview
Figure 1.1.1a shows the number of peer-reviewed AI
publications, and Figure 1.1.1b shows the share of thoseamong all peer-reviewed publications in the world. The
total number of publications grew by nearly 12 times
between 2000 and 2019. Over the same period, thepercentage of peer-reviewed publications increased from
0.82% of all publications in 2000 to 3.8% in 2019.

By Region1
Among the total number of peer-reviewed AI publicationsin the world, East Asia & Pacific has held the largest sharesince 2004, followed by Europe & Central Asia, and North
America (Figure 1.1.2). Between 2009 and 2019, South Asiaand sub-Saharan Africa experienced the highest growthin terms of the number of peer-reviewed AI publications,increasing by eight- and sevenfold, respectively.

NUMBER of PEER-REVIEWED AI PUBLICATIONS, 2000-19

120

100

80

60

40

20

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0
2000

Number of Peer-Reviewed AI Publications (in Thousands)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Figure 1.1.1a
1 Regions in this chapter are classified according to the World Bank analytical grouping.

CHAPTER 1 PREVIEW

5CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

PEER-REVIEWED AI PUBLICATIONS (% of TOTAL), 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index ReportPeer-Reviewed AI Publications (% of Total)

4%

3.8%

3%

2%

1%

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.1b

PEER-REVIEWED AI PUBLICATIONS (% of TOTAL) by REGION, 2000-19
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index ReportPeer-Reviewed AI Publications (% of Total)

40%
36.9% East Asia & Pacific30%
25.1% Europe & Central Asia

20%
17.0% North America

10%8.8% South Asia
5.5% Middle East & North Africa
2.7% Latin America & Caribbean

2023

2022

2021

2020

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

2019

0.7% Sub-Saharan Africa

0%Figure 1.1.2

CHAPTER 1 PREVIEW

6

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
To compare the activity among the world’s major AI
players, this section shows trends of peer-reviewed AI
publications coming out of China, the European Union,
and the United States. As of 2019, China led in the shareof peer-reviewed AI publications in the world, after

overtaking the European Union in 2017 (Figure 1.1.3).
It published 3.5 times more peer-reviewed AI papers
in 2019 than it did in 2014—while the European Union
published just 2 times more papers and the United States
2.75 times more over the same period.

PEER-REVIEWED AI PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-19

Peer-Reviewed AI Publications (% of World Total)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report25%
22.4% China

20%

16.4% EU

15%
14.6% US

10%

5%

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.3

CHAPTER 1 PREVIEW

7

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Institutional Affiliation
The following charts show the number of peer-reviewed
AI publications affiliated with corporate, government,
medical, and other institutions in China (Figure 1.1.4a),the European Union (Figure 1.1.4b), and the United States(Figure 1.1.4c).2 In 2019, roughly 95.4% of overall peerreviewed AI publications in China were affiliated with theacademic field, compared with 81.9% in the European
Union and 89.6% in the United States. Those affiliation
categories are not mutually exclusive, as some authors

could be affiliated with more than one type of institution.
The data suggests that, excluding academia, government
institutions—more than those in other categories—
consistently contribute the highest percentage of peerreviewed AI publications in both China and the European
Union (15.6% and 17.2 %, respectively, in 2019), whilein the United States, the highest portion is corporateaffiliated (19.2%).

NUMBER of PEER-REVIEWED AI PUBLICATIONS in CHINA by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

4,000

3,000

2,000

1,675 Corporate

1,000
382 Medical
14 Other

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

2019

Number of Peer-Reviewed AI Publications

4,352 Government

Figure 1.1.4a

2 Across all three geographic areas, the number of papers affiliated with academia exceeds that of government-, corporate-, and medical-affiliated ones; therefore, the academia affiliation is not shown,as it would distort the graphs.

CHAPTER 1 PREVIEW

8

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

NUMBER of PEER-REVIEWED AI PUBLICATIONS in the EUROPEAN UNION by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Number of Peer-Reviewed AI Publications

3,523 Government

3,000

2,000
1,594 Corporate

1,000
508 Medical
187 Other

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.4b

NUMBER of PEER-REVIEWED AI PUBLICATIONS in the UNITED STATES by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

Number of Peer-Reviewed AI Publications

3,513 Corporate

3,000

2,277 Gov.

2,000

1,000

718 Medical

120 Other

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.4c

CHAPTER 1 PREVIEW

9CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

Academic-Corporate Collaboration
Since the 1980s, the R&D collaboration between
academia and industry in the United States has grownin importance and popularity, made visible by the
proliferation of industry-university research centers aswell as corporate contributions to university research.
Figure 1.1.5 shows that between 2015 and 2019, theUnited States produced the highest number of hybrid
academic-corporate, co-authored, peer-reviewed AI
publications—more than double the amount in theEuropean Union, which comes in second, followed by
China in third place.

NUMBER of ACADEMIC-CORPORATE PEER-REVIEWED AI PUBLICATIONS by GEOGRAPHIC AREA, 2015-19 (SUM)
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report

United States
European Union
China
United Kingdom
Germany
Japan
FranceCanadaSouth Korea
Netherlands
Switzerland
India
Hong Kong
Spain
Italy
0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

Number of Peer-Reviewed AI Publications
Figure 1.1.5

CHAPTER 1 PREVIEW

10CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

according to the world average. For example, an FWCI of
0.75 means 25% fewer citations than the world average.

To assess how academic-corporate collaborations
impact the Field-Weighted Citation Impact (FWCI) of
AI publications from different geographic regions,
see Figure 1.1.6. FWCI measures how the number ofcitations received by publications compares with theaverage number of citations received by other similar
publications in the same year, discipline, and format(book, article, conference paper, etc.). A value of 1.0
represents the world average. More than or less than 1means publications are cited more or less than expected,

The chart shows the FWCI for all peer-reviewed AI
publications on the y-axis and the total number (on a log
scale) of academic-corporate co-authored publications
on the x-axis. To increase the signal-to-noise ratio of the
FWCI metric, only countries that have more than 1,000
peer-reviewed AI publications in 2020 are included.

PEER-REVIEWED AI PUBLICATIONS' FIELD-WEIGHTED CITATION IMPACT and NUMBER of ACADEMIC-CORPORATE
PEER-REVIEWED AI PUBLICATIONS, 2019
Peer-Reviewed AI Publications' Field-Weighted Citation Impact (FWCI)

Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report3

Singapore

Hong Kong

Australia

Switzerland

Netherlands
2

Iran

CanadaMalaysia

Italy
Spain

GermanySouth Korea
European Union

FranceTurkeyRussiaIndonesia

China

Japan

Taiwan1

BrazilUnited States

United Kingdom

India

0
5

10

20

50

100

200

500

1,000

2,000

Number of Academic-Corporate Peer-Reviewed AI Publications (Log Scale)Figure 1.1.6

CHAPTER 1 PREVIEW

11

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I J O U R N A L P U B L I CAT I O N S

Overview

The next three sections chart the trends in the publicationof AI journals, conference publications, and patents, aswell as their respective citations that provide a signal
for R&D impact, based on data from Microsoft Academic
Graph. MAG3 is a knowledge graph consisting of more than
225 million publications (at the end of November 2019).

Overall, the number of AI journal publications in 2020 is 5.4
times higher than it was in 2000 (Figure 1.1.7a). In 2020, thenumber of AI journal publications increased by 34.5% from2019—a much higher percentage growth than from 2018 to2019 (19.6%). Similarly, the share of AI journal publications
among all publications in the world has jumped by 0.4
percentage points in 2020, higher than the average of 0.03
percentage points in the past five years (Figure 1.1.7b).

NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Journal Publications (in Thousands)

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.7a

AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20

3%

2.2%

2%

1%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Journal Publications (% of All Publications)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.7b
3 See “An Overview of Microsoft Academic Service (MAS) and Applications” and “A Review of Microsoft Academic Services for Science of Science Studies” for more details.

CHAPTER 1 PREVIEW

12

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Region
Figure 1.1.8 shows the share of AI journals—the dominantpublication entity in terms of numbers in the MAG
database—by region between 2000 and 2020. East Asia
& Pacific, Europe & Central Asia, and North America are
responsible for the majority of AI journal publications in
the past 21 years, while the lead position among the three

regions changes over time. In 2020, East Asia & Pacific held
the highest share (26.7%), followed by Europe & Central
Asia (13.3%) and North America (14.0%). Additionally,in the last 10 years, South Asia, and Middle East & North
Africa saw the most significant growth, as the number of
AI journal publications in those two regions grew six- andfourfold, respectively.

AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report30%
26.7%, East Asia & Pacific20%
14.0%, North America
13.3%, Europe & Central Asia

10%4.9%, South Asia

2024

2023

2022

2021

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%

2020

3.1%, Middle East & North Africa
1.3%, Latin America & Caribbean
0.3%, Sub-Saharan Africa

2000

AI Journal Publications (% of World Total)

40%

Figure 1.1.8

CHAPTER 1 PREVIEW

13

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
Figure 1.1.9 shows that among the three major AI powers, China has had the largest share of AI journal publications in theworld since 2017, with 18.0% in 2020, followed by the United States (12.3%) and the European Union (8.6%).
AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
AI Journal Publications (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%

18.0%, China

15%
12.3%, US

10%8.6%, EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.9

Citation

In terms of the highest share of AI journal citations, Figure 1.1.10 shows that China (20.7%) overtook the United States(19.8%) in 2020 for the first time, while the European Union continued to lose overall share.
AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Citations (% of World Total)

40%

30%

20.7%
China

20%

19.8%
US

10%

11.0%
EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.10

CHAPTER 1 PREVIEW

14

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I C O N F E R E N C E P U B L I CAT I O N S

Overview

Between 2000 and 2019, the number of AI conference publications increased fourfold, although the growth flattenedout in the past ten years, with the number of publications in 2019 just 1.09 times higher than the number in 2010.4
NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of Publications (in Thousands)

50

40

30

20

10

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.11a

AI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of All Publications)

25%
20.2%

20%

15%

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.11b
4 Note that conference data in 2020 on the MAG system is not yet complete. See the Appendix for details.

CHAPTER 1 PREVIEW

15

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Region
Asia & Pacific took the lead starting in 2004, accounting
for more than 27% in 2020. North America overtook
Europe & Central Asia to claim second place in 2018,accounting for 20.1%, followed by 21.7% in 2020.

Figure 1.1.12 shows that, similar to the trends in AI
journal publication, East Asia & Pacific, Europe & Central
Asia, and North America are the world’s dominantsources for AI conference publications. Specifically, East

AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report30%
27.3%, East Asia & Pacific21.7%, North America

20%

18.6%, Europe & Central Asia

10%

2026

2025

2024

2023

2022

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

1999

2000

0%

2021

5.1%, South Asia
2.2%, Middle East & North Africa
1.7%, Latin America & Caribbean
0.3%, Sub-Saharan Africa

2020

AI Conference Publications (% of World Total)

40%

Figure 1.1.12

CHAPTER 1 PREVIEW

16

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
is almost nine times higher than it was in 2000. The
share of conference publications for the European Union
peaked in 2011 and continues to decline.

China overtook the United States in the share of AI
conference publications in the world in 2019 (Figure
1.1.13). Its share has grown significantly since 2000.
China’s percentage of AI conference publications in 2019

AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20

AI Conference Publications (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%

19.4% US15%

15.2% China
12.8% EU

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.1.13

Citation

21 years. The United States tops the list with 40.1% of
overall citations in 2020, followed by China (11.8%) andthe European Union (10.9%).

With respect to citations of AI conference publications,Figure 1.1.14 shows that the United States has held adominant lead among the major powers over the past

AI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
40.1% US

40%

30%

20%
11.8% China

10%

10.9% EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Conference Citations (% of World Total)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.14

CHAPTER 1 PREVIEW

17

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A I PAT E N T S
Overview
The total number of AI patents published in the world
has been steadily increasing in the past two decades,
growing from 21,806 in 2000 to more than 4.5 times that,or 101,876, in 2019 (Figure 1.1.15a). The share of AI patents
published in the world exhibits a lesser increase, fromNUMBER of AI PATENT PUBLICATIONS, 2000-20

around 2% in 2000 to 2.9% in 2020 (Figure 1.1.15b). The AI
patent data is incomplete—only 8% of the dataset in 2020
includes a country or regional affiliation. There is reason
to question the data on the share of AI patent publications
by both region and geographic area, and it is therefore notincluded in the main report. See the Appendix for details.

Number of AI Patent Publications (in Thousands)

Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

100

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.1.15a

AI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 2000-20
2.9%

3%

2%

1%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

0%
2000

AI Patent Publications (% of All Publications)

Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Figure 1.1.15b

CHAPTER 1 PREVIEW

18

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

A R X I V P U B L I CAT I O N S

Overview

In addition to the traditional avenues for
publishing academic papers (discussed
above), AI researchers have embraced thepractice of publishing their work (often
pre–peer review) on arXiv, an online
repository of electronic preprints. arXiv
allows researchers to share their findings
before submitting them to journals andconferences, which greatly accelerates
the cycle of information discovery anddissemination. The number of AI-related
publications in this section includes
preprints on arXiv under cs.AI (artificial
intelligence), cs.CL (computation andlanguage), cs.CV (computer vision), cs.NE(neural and evolutionary computing),
cs.RO (robotics), cs.LG (machine learning incomputer science), and stat.ML (machinelearning in statistics).

In just six years, the number of AI-related publications on arXiv grew
more than sixfold, from 5,478 in 2015 to 34,736 in 2020 (Figure 1.1.16).
NUMBER of AI-RELATED PUBLICATIONS on ARXIV, 2015-20
Number of AI-Related Publications on arXiv (in Thousands)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

30

20

10

0
2015

2016

2017

2018

By Region

2019
Figure 1.1.16

The analysis by region shows that while North America still holds the lead in the global share of arXiV AI-related
publications, its share has been decreasing—from 41.6% in 2017 to 36.3% in 2020 (Figure 1.1.17). Meanwhile, the share of
publications in East Asia & Pacific has grown steadily in the past five years—from 17.3% in 2015 to 26.5% in 2020.
ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by REGION, 2015-20
AI-Related Publications on arXiv (% of World Total)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

40%
36.3% North America

30%
26.5% East Asia & Paci c
22.9% Europe & Central Asia

20%

10%
4.0% South Asia
2.5% Middle East & North Africa
1.3% Latin America & Caribbean
0.3% Sub-Saharan Africa

0%
2015

CHAPTER 1 PREVIEW

2016

2017

2018

2019

2020

2021

Figure 1.1.17

19

2020

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

By Geographic Area
While the total number of AI-related publications on arXiv
is increasing among the three major AI powers, China iscatching up with the United States (Figure 1.1.18a andFigure 1.1.18b). The share of publication counts by theEuropean Union, on the other hand, has remained largely
unchanged.

NUMBER of AI-RELATED PUBLICATIONS on ARXIV by GEOGRAPHIC AREA, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
11,280US

Number of AI-Related Publications on arXiv

12,00010,000

8,000
6,505
EU

6,000

5,440
China

4,000

2,000

0
2015

2016

2017

2018

2019

2020
Figure 1.1.18a

ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2015-20

AI-Related Publications on arXiv (% of World Total)

Source: arXiv, 2020 | Chart: 2021 AI Index Report

32.5% US30%

20%

18.7% EU
15.7% China

10%

0%
2015

2016

2017

2018

2019

2020
Figure 1.1.18b

CHAPTER 1 PREVIEW

20

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

By Field of Study
Among the six fields of study related to AI on arXiv,
the number of publications in Robotics (cs.RO) andMachine Learning in computer science (cs.LG) haveseen the fastest growth between 2015 and 2020,
increasing by 11 times and 10 times respectively(Figure 1.1.19). In 2020, cs.LG and Computer Vision(cs.CV) lead in the overall number of publications,accounting for 32.0% and 31.7%, respectively, of all AIrelated publications on arXiv. Between 2019 and 2020,
the fastest-growing categories of the seven studied
here were Computation and Language (cs.CL), by
35.4%, and cs.RO, by 35.8%.

1.1 P U B L I CAT I O N S

Among the six fields ofstudy related to AI on arXiv,
the number of publications
in Robotics (cs.RO) andMachine Learning incomputer science (cs.
LG) have seen the fastest
growth between 2015 and2020, increasing by 11 times
and 10 times respectively.

NUMBER of AI-RELATED PUBLICATIONS on ARXIV by FIELD of STUDY 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
11,001 cs.CV

Number of AI-Related Publications on arXiv

11,098 cs.LG

10,000

8,000

6,000

5,573 cs.CL

4,000
2,571 cs.RO

2,000

1,923 cs.AI
1,818 stat.ML
743 cs.NE

0
2015

2016

2017

2018

2019

2020
Figure 1.1.19

CHAPTER 1 PREVIEW

21

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1.1 P U B L I CAT I O N S

Deep Learning Papers on arXiv
With increased access to data and significant
improvements in computing power, the fieldof deep learning (DL) is growing at breakneck
speed. Researchers from Nesta used a topic
modeling algorithm to identify the deep learning
papers on arXiv by analyzing the abstract ofarXiv papers under the Computer Science (CS)and Machine Learning in Statistics (state.ML)
categories. Figure 1.1.20 suggests that in thelast five years alone, the overall number of DL
publications on arXiv grew almost sixfold.

NUMBER of DEEP LEARNING PUBLICATIONS on ARXIV, 2010-19
Number of Deep Learning Publications on arXiv (in Thousands)

Source: arXiv/Nesta, 2020 | Chart: 2021 AI Index Report

7

6

5

4

3

2

1
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

Figure 1.1.20

CHAPTER 1 PREVIEW

22

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

1. 2 C O N F E R E N C E S

Conference attendance is an indication of broader industrial and academic interest in a scientific field. In the past 20 years,
AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends inattendance at and submissions to major AI conferences.

1.2 CONFERENCESC O N F E R E N C E AT T E N DA N C E
Last year saw a significant increase in participation levels
at AI conferences, as most were offered through a virtual
format. Only the 34th Association for the Advancementof Artificial Intelligence (AAAI) Conference on Artificial
Intelligence was held in person in February 2020.
Conference organizers report that a virtual format allows
for higher attendance of researchers from all over theworld, though exact attendance numbers are difficult tomeasure.
Due to the atypical nature of 2020 conference attendance
data, the 11 major AI conferences in 2019 have been
split into two categories based on 2019 attendance data:
large AI conferences with over 3,000 attendees and small
AI conferences with fewer than 3,000 attendees. Figure
1.2.1 shows that in 2020, the total number of attendees
across nine conferences almost doubled.5 In particular,
the International Conference on Intelligent Robots
and Systems (IROS) extended the virtual conference
to allow users to watch events for up to three months,
which explains the high attendance count. Because theInternational Joint Conference on Artificial Intelligence(IJCAI) was held in 2019 and January 2021—but not in
2020—it does not appear on the charts.

Conference organizers
report that a virtual
format allows for higher
attendance of researchers
from all over the world,though exact attendance
numbers are difficult tomeasure.

5 For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference; For the KR conference,
the attendance in 2020 is based on the number of registrations; For the ICPAS conference, the attendance of 450 in 2020 is an estimate as some participants may have used anonymous Zoom accounts.

CHAPTER 1 PREVIEW

23

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 2 C O N F E R E N C E SATTENDANCE at LARGE AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report

25,719 IROS

25,00022,011 NeurIPS

Number of Attendees

20,00015,00010,800 ICML

10,0007,500 CVPR

5,000
4,884 AAAI

2020

3,015 IJCAI

2019

2017

2016

2015

2014

2013

2012

2011

2010

0

2018

3,050 ICRA

Figure 1.2.1

ATTENDANCE at SMALL AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report
5,600
ICLR

Number of Attendees

5,000
3,972
ACL

4,000

3,726
AAMAS

3,000

2,000

1,000

469
KR
450
ICAPS

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 1.2.2

CHAPTER 1 PREVIEW

24

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 2 C O N F E R E N C E S

Corporate Representation at AI Research Conferences
Researchers from Virginia Tech and Ivey
Business School, Western University foundthat large technology firms have increased
participation in major AI conferences. In
their paper, titled “The De-Democratizationof AI: Deep Learning and the Compute
Divide in Artificial Intelligence Research,” theressearchers use the share of papers affiliatedwith firms over time at AI conferences toillustrate the increased presence of firmsin AI research. They argue that the unequal

distribution of compute power in academia,
which they refer to as the “compute divide,”is adding to the inequality in the era of deeplearning. Big tech firms tend to have moreresources to design AI products, but they also
tend to be less diverse than less elite or smaller
institutions. This raises concerns about bias andfairness within AI. All 10 major AI conferences
displayed in Figure 1.2.3 show an upward trend
in corporate representation, which further
extends the compute divide.

SHARE of FORTUNE GLOBAL 500 TECH-AFFILIATED PAPERS
Source: Ahmed & Wahed, 2020 | Chart: 2021 AI Index Report40%

30.8% KDD

% of Fortune Global 500 Tech-Affiliated Papers

30%

29.0% NeurIPS

28.7% ACL

25.6% ECCV

23.7% ICCV

28.5% EMNLP

20%
10%
0%
40%

27.9% ICML

30%

21.9% CVPR

20%
10%
0%
40%
30%
19.3% AAAI

20%

17.5% IJCAI

10%
0%

2000

2019

2000

2019

2000

2019

2000

2019
Figure 1.2.3

CHAPTER 1 PREVIEW

25

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

1. 3 A I O P E N S O U R C E S O F T WA R E
LIBRARIESA software library is a collection of computer code that is used to create applications and products. Popular AI-specific software
libraries—such as TensorFlow and PyTorch—help developers create their AI solutions quickly and efficiently. This section analyzes
the popularity of software libraries through GitHub data.

1.3 AI OPEN-SOURCESOFTWARE LIBRARIESG I T H U B S TA R S
GitHub is a code hosting platform that AI researchers anddevelopers frequently use to upload, comment on, anddownload software. GitHub users can “star” a projectto save it in their list, thereby expressing their interestsand likes—similar to the “like’’ function on Twitter andother social media platforms. As AI researchers upload
packages on GitHub that mention the use of an opensource library, the “star” function on GitHub can be used
to measure the popularity of various AI programming
open-source libraries.
Figure 1.3.1 suggests that TensorFlow (developed by
Google and publicly released in 2017) is the most popular
AI software library. The second most popular library in2020 is Keras (also developed by Google and built on topof TensorFlow 2.0). Excluding TensorFlow, Figure 1.3.2
shows that PyTorch (created by Facebook) is anotherlibrary that is becoming increasingly popular.

CHAPTER 1 PREVIEW

TensorFlow (developedby Google and publicly
released in 2017) is themost popular AI software
library. The second mostpopular library in 2020 is
Keras (also developed by
Google and built on topof TensorFlow 2.0).

26

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Artificial IntelligenceIndex Report 2021

1. 3 A I O P E N S O U R C E S O F T WA R E
LIBRARIESNUMBER of GITHUB STARS by AI LIBRARY, 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
153 TensorFlow

Cumulative Github Stars (in Thousand)

150

100

51 Keras

50

46 PyTorch
45 Sckit-learn
31 BVLC/caffe
19 MXNet
17 Cntk
9 Theano
8 Caffe2

0
2014

2015

2016

2017

2018

2019

2020
Figure 1.3.1

NUMBER of GITHUB STARS by AI LIBRARY (excluding TENSORFLOW), 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
51 Keras

Cumulative Github Stars (in Thousand)50

46 PyTorch
45 Sckit-learn

40

31 BVLC/caffe

30

19 MXNet

20

17 Cntk

10

9 Theano
8 Caffe2

0
2014

2015

2016

2017

2018

2019

2020
Figure 1.3.2

CHAPTER 1 PREVIEW

27

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXELSEVIER
Prepared by Jörg Hellwig and Thomas A. Collins

SourceElsevier’s Scopus database of scholarly publications has
indexed more than 81 million peer-reviewed documents.
This data was compiled by Elsevier.

Methodology
Scopus tags its papers with keywords, publication dates,
country affiliations, and other bibliographic information.
The Elsevier AI Classifier leveraged the following features
extracted from the Scopus records that were returned asa result of querying against the provided approximately
800 AI search terms. Each record fed into the feature
creation also maintained a list of each search term that
hit for that particular record:• hasAbs: Boolean value whether or not the record had an
abstract text section in the record (e.g., some records
are only title and optional keywords)• coreCnt: number of core-scored search terms present
for the record• mediumCnt: number of medium-scored search terms
present for the record• lowCnt: number of low-scored search terms present for
the record• totalCnt: total number of search terms present for therecord• pcntCore: coreCnt/totalCnt• pcntMedium: mediumCnt/totalCnt• pcntLow: lowCnt/totalCnt• totalWeight = 5*coreCnt + 3*mediumCnt + 1*lowCnt• normWeight = if (has Abs) { totalWeight / (title.length +abstract.length) } else• { totalWeight/title.length}• hasASJC: Boolean value: does the record have anassociated ASJC list?
• isAiASJC: does ASJC list contain 1702?

CHAPTER 1 PREVIEW• isCompSciASJC does ASJC list contain a 17XX ASJC code(“1700,” “1701,” “1702,” “1703,” “1704,” “1705,” “1706,”“1707,” “1708,” “1709,” “1710,” “1711,” “1712”)• isCompSubj: does the Scopus record have a
ComputerScience subject code associated with it? Thisshould track 1:1 to isCompSciASJC. Scopus has 27 major
subject areas of which one is Computer Science. The
feature checks, if the publication is within Computer
Science or not. This is no exclusion.pcntCompSciASJC:
percentage of ASJC codes for record that are from the
CompSci ASJC code list
Details on Elsevier’s dataset defining AI, country
affiliations, and AI subcategories can be found in the 2018
AI Index Report Appendix.

Nuance• The Scopus system is retroactively updated. As a result,the number of papers for a given query may increaseover time.
• Members of the Elsevier team commented that data on
papers published after 1995 would be the most reliable.
The raw data has 1996 as the starting year for Scopus data.

Nuances specific to AI publications by region• Papers are counted utilizing whole counting rather
than fractional counting. Papers assigned to multiple
countries (or regions) due to collaborations are counted
toward each country (or region). This explains why topline numbers in a given year may not match individual
country numbers. For example, a paper assigned toGermany, France, and the United States will appear oneach country’s count, but only once for Europe (plusonce for the U.S.) as well as being counted only at the
global level.
• “Other” includes all other countries that have published
one or more AI papers on Scopus.

28Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

Nuances specific to publications by topic• The 2017 AI Index Report showed only AI papers within
the CS category. In the 2018 and 2019 reports, all papers
tagged as AI were included, regardless of whether theyfell into the larger CS category.
• Scopus has a subject category called AI, which is asubset of CS, but this is relevant only for a subjectcategory approach to defining AI papers. The
methodology used for the report includes all papers,
since increasingly not all AI papers fall under CS.

Nuances specific to methodology• The entire data collection process was done by Elsevier
internally. The AI Index was not involved in the keyword
selection process or the counting of relevant papers.
• The boundaries of AI are difficult to establish, in part
because of the rapidly increasing applications inmany fields, such as speech recognition, computer
vision, robotics, cybersecurity, bioinformatics, andhealthcare. But limits are also difficult to define becauseof AI’s methodological dependency on many areas,such as logic, probability and statistics, optimization,
photogrammetry, neuroscience, and game theory—toname just a few. Given the community’s interest in AI
bibliometrics, it would be valuable if groups producing
these studies strived for a level of transparency in their
methods, which would support the reproducibilityof results, particularly on different underlying
bibliographic databases.

AI Training Set
A training set of approximately 1,500 publications
defines the AI field. The set is only the EID (the Scopus
identifier of the underlying publications). Publications
can be searched and downloaded either from Scopus
directly or via the API.The training set is a set of
publications randomly selected from the initial sevenmio publications. After running the algorithm we verify
the results of the training set with the gold set (expert
hand-checked publications which are definitely AI).

CHAPTER 1 PREVIEW

29

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

 I C R O S O F T ACA D E M I C G R A P H :M
M E T H O D O LO GY
Prepared by Zhihong Shen, Boya Xie, Chiyuan Huang,
Chieh-Han Wu, and Kuansan Wang

SourceThe Microsoft Academic Graph1 is a heterogeneous graphcontaining scientific publication records and citation
relationships between those publications, as well as
authors, institutions, journals, conferences, and fields ofstudy. This graph is used to power experiences in Bing,
Cortana, Word, and Microsoft Academic. The graph iscurrently being updated on a weekly basis. Learn more
about MAG here.

Methodology
MAG Data Attribution: Each paper is counted exactly
once. When a paper has multiple authors or regions,the credit is equally distributed to the unique regions.
For example, if a paper has two authors from theUnited States, one from China, and one from the United
Kingdom, then the United States, China, and the United
Kingdom each get one-third credit.
Metrics: Total number of published papers (journal
papers, conference papers, patents, repository2); total
number of citations of published papers.
Definition: The citation and reference countrepresents the number of respective metrics for AI
papers collected from all papers. For example, in“OutAiPaperCitationCountryPairByYearConf.csv,” a row
stating “China, United States, 2016, 14955” means that
China’s conference AI papers published in 2016 received
14,955 citations from (all) U.S. papers indexed by MAG.
Curating the MAG Dataset and References: Generally
speaking, the robots sit on top of a Bing crawler to readeverything from the web and have access to the entire
web index. As a result, MAG is able to program the robots

to conduct more web searches than a typical humancan complete. This helps disambiguate entities withthe same names. For example, for authors, MAG gets toadditionally use all the CVs and institutional homepages
on the web as signals to recognize and verify claims3.
MAG has found this approach to be superior to the resultsof the best of the KDD Cup 2013 competition, which uses
only data from within all publication records and Open
Researcher and Contributor Identifiers (ORCIDs).

Notes About the MAG Data
Conference Papers: After the contents and data sourceswere scrutinized, it was determined that some of the 2020
conference papers were not properly tagged with theirconference venue. Many conference papers in the MAG
system are under arXiv papers, but due to issues arisingfrom some data sources (including delays in DBLP andweb form changes on the ACM website), they were possibly
omitted as 2020 conference papers (ICML-PKDD, IROS, etc.).
However, the top AI conferences (selected not in terms of
publication count, but rather considering both publicationand citation count as well as community prestige) arecomplete. In 2020, the top 20 conferences presented 103,000
papers, which is 13.7% of all AI conference papers, and theyreceived 7.15 million citations collectively, contributing
47% of all citations received for all AI conference papers.
The number of 2020 conference publications is slightly
lower than in 2019. Data is known to be missing for ICCVand NAACL. About 100 Autonomous Agents and Multiagent
Systems (AAMAS) conference papers are erroneously
attributed to an eponymous journal.
Unknown Countries for Journals and Conferences:
For the past 20 to 30 years, 30% of journal and conference
affiliation data lacks affiliation by country or region, due
to errors in paper format, data source, and PDF parsing,among others.

1 See “A Review of Microsoft Academic Services for Science of Science Studies” and “Microsoft Academic Graph: When Experts Are Not Enough” for more details.
2 Repository as a publication type in MAG refers to both preprints and postprints. In the AI domain, it predominantly comes from arXiv. See “Is Preprint the Future of Science? A Thirty Year Journey of
Online Preprint Services” for details.
3 See “Machine Verification for Paper and Author Claims” and “How Microsoft Academic Uses Knowledge to Address the Problem of Conflation/Disambiguation” for details.

CHAPTER 1 PREVIEW

30

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXM I C R O S O F T ACA D E M I C G R A P H :PAT E N T DATA C H A L L E N G E
As mentioned in the report, the patent data—especiallythe affiliation information—is incomplete in the MAG
database. The reason for the low coverage is twofold.
First, applications published by the patent offices often
identify the inventors by their residencies not affiliations.
While patent applications often have the information
about the “assignees” of a patent, they do not necessarilymean the underlying inventions originate from theassignee institutions. Therefore, detected affiliations may
be inaccurate. In case a patent discloses the scholarly
publications underlying the invention, MAG can infer
inventors’ affiliations through the scholarly publications.
Second, to maximize intellectual property protection
around the globe, institutions typically file multiple
patent applications on the same invention under various

jurisdictions. These multiple filings, while appear very
different because the titles and inventor names are oftentranslated into local languages, are in fact the result of asingle invention. Raw patent counts therefore inflate theinventions in their respective domains. To remediate thisissue, MAG uses the patent family ID feature to combine allfilings with the original filing, which allows the database
to count filings all around the world of the same origin
only once.4 Conflating the multiple patent applications of
the same invention is not perfect, and over-conflations ofpatents are more noticeable in MAG than scholarly articles.
These challenges raise questions about the reliability ofdata on the share of AI patent publications by both region
and geographic area. Those charts are included below.

By Region
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20

AI Patent Publications (% of World Total)

Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

8%

6%

4%
3.2%, North America
2.6%, East Asia & Pacific2%

1.6%, Europe & Central Asia
0.1%, South Asia
0.0%, Sub-Saharan Africa
0.0%, Latin America & Caribbean
0.0%, Middle East & North Africa

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.1

4 Read “Sharpening Insights into the Innovation Landscape with a New Approach to Patents” for more details.

CHAPTER 1 PREVIEW

31

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

By Geographic Area
AI PATENT PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of World Total)

8%

6%

4%
3.2% US2%

1.3% EU
0.4% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.2

Citation
AI PATENT CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Citations (% of World Total)

25%

20%

15%

10%

8.6% US5%2.1% EU
2.0% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.4.3

CHAPTER 1 PREVIEW

32

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

M I C R O S O F T ACA D E M I C G R A P H :MEASUREMENT CHALLENGES AND
A LT E R N AT I V E D E F I N I T I O N O F A I
As the AI Index team discussed in the paper “Measurementin AI Policy: Opportunities and Challenges,” choosing how
to define AI and correctly capture relevant bibliometricdata remain challenging. Data in the main report is basedon a restricted definition of AI, adopted by MAG, that aligns
with what has been used in previous AI Index reports.
One consequence is that such a definition excludes many
AI publications from venues considered to be core AI
venues. For example, only 25% of conference publicationsin the 2020 AAAI conference are included in the original
conference dataset.

papers in the MAG database tagged with a field of study
that is categorized in either the “artificial intelligence”
or the “machine learning” field of study as well as theirsubtopics in the MAG taxonomy.5 This is a more liberal
definition than the one used by MAG, which considers only
those publications tagged with “artificial intelligence”
as AI publications. For example, an application paper inbiology that uses ML techniques will be counted as an AI
publication under the OECD definition, but not under the
MAG definition unless the paper is specifically tagged in
the AI category.

To spur discussion on this important topic, this section
presents the MAG data with an alternative definition of AI
used by the Organisation for Economic Co-operation andDevelopment (OECD). OECD defines AI publications as

Charts corresponding to those in the main text but using
the OECD definition are presented below. The overall
trends are very similar.

AI Journal Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Journal Publications (in Thousands)

300

250

200

150

100

50

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.1a

5 Read the OECD.AI Policy Observatory MAG methodological note for more details on the MAG-OECD definition of AI and “A Web-scale System for Scientific Knowledge Exploration” on the MAG Taxonomy.

CHAPTER 1 PREVIEW

33

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
7.9%

AI Journal Publications (% of All Publications)

8%

6%

4%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2%

Figure 1.5.1b

OECD DEFINITION: AI JOURNAL PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

25%

20%
16.5% Europe & Central Asia
16.2% North America

15%

10%

5%

4.2% South Asia
4.1% Middle East & North Africa

2025

2024

2023

2022

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2021

2.2% Latin America & Caribbean
1.0% Sub-Saharan Africa

0%

2020

AI Journal Publications (% of World Total)

23.0% East Asia & PacificFigure 1.5.2

CHAPTER 1 PREVIEW

34

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI JOURNAL PUBLICATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Publications (% of World Total)

20%

14.4%
China

15%

14.1%
US

10%

10.8%
EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.3

OECD DEFINITION: AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Journal Publications (% of World Total)

20%

15%

10%

8.4%
China
8.2%
US
5.8%
EU

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.4

CHAPTER 1 PREVIEW

35

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

AI Conference Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of Publications (in Thousands)

100

80

60

40

20

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.5aOECD DEFINITION: AI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of All Publications)

40%

37.6%

35%

30%

25%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

20%

Figure 1.5.5b

CHAPTER 1 PREVIEW

36

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXOECD DEFINITION: AI CONFERENCE PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of World Total)

40%

30%
26.5% East Asia & Pacific21.2% North America

20%

20.0% Europe & Central Asia

10%5.0% South Asia
2.4% Middle East & North Africa
1.8% Latin America & Caribbean

0%

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.4% Sub-Saharan Africa

Figure 1.5.6

OECD DEFINITION: AI CONFERENCE PUBLICATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Conference Publications (% of World Total)

25%

18.9%
US

20%

15.0%
China

15%
14.2%
EU

10%

5%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.7

CHAPTER 1 PREVIEW

37

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI CONFERENCE CITATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

14.3%
US

AI Conference Citation (% of World Total)

15%

10%

4.2%
China

5%

4.2%
EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.8

AI Patent Publications (OECD Definition)
OECD DEFINITION: NUMBER of AI PATENT PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

Number of AI Patent Publications (in Thousands)

200

150

100

50

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0

Figure 1.5.9a

CHAPTER 1 PREVIEW

38

Artificial IntelligenceIndex Report 2021

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

APPENDIXOECD DEFINITION: AI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of All Publications)

5%

4.6%

4%

3%

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

2%

Figure 1.5.9b

OECD DEFINITION: AI PATENT PUBLICATION (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

8%

6%

4%
3.3% North America
2.1% East Asia & Pacific2%

1.4% Europe & Central Asia
0.1% Middle East & North Africa
0.1% South Asia
0.0% Sub-Saharan Africa

0%

2024

2023

2022

2021

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.0% Latin America & Caribbean

2020

AI Patent Publications (% of World Total)

10%

Figure 1.5.10

CHAPTER 1 PREVIEW

39

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

OECD DEFINITION: AI PATENT PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Publications (% of World Total)

10%

8%

6%

4%

3.2% US2%
1.0% EU
0.4% China

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.11

OECD DEFINITION: AI PATENT CITATION (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report

AI Patent Citation (% of World Total)

25%

20%

15%

10%
5.6% US5%
1.2% China
1.1% EU

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0%

Figure 1.5.12

CHAPTER 1 PREVIEW

40

Artificial IntelligenceIndex Report 2021

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

PA P E R S O N A R X I V
Prepared by Jim Entwood and Eleonora Presani

SourcearXiv.org is an online archive of research articles in
the fields of physics, mathematics, computer science,quantitative biology, quantitative finance, statistics,
electrical engineering and systems science, andeconomics. arXiv is owned and operated by Cornell
University. See more information on arXiv.org.

Methodology
Raw data for our analysis was provided byrepresentatives at arXiv.org. The keywords we selected,
and their respective categories, are below:
Artificial intelligence (cs.AI)
Computation and language (cs.CL)
Computer vision and pattern recognition (cs.CV)
Machine learning (cs.LG)
Neural and evolutionary computing (cs.NE)
Robotics (cs.RO)
Machine learning in stats (stats.ML)

arXiv is actively looking at ways to improve how it canbetter support AI/ML researchers as the field grows
and discovering content becomes more challenging.
For example, there may be ways to create finer grained
categories in arXiv for machine learning to help
researchers in subfields share and find work more easily.
The other rapidly expanding area is computer vision,
where there is considerable overlap for ML applicationsof computer vision.

Nuance• Categories are self-identified by authors—those shown
are selected as the “primary” category. Thus there is nota single automated categorization process. Additionally,
the artificial intelligence or machine learning categories
may be categorized by other subfields or keywords.
• arXiv team members suggest that participation onarXiv can breed greater participation, meaning that anincrease in a subcategory on arXiv could drive overindexed participation by certain communities.

For most categories, arXiv provided data for 2015–2020.
To review other categories’ submission rates on arXiv, seearXiv.org’s submission statistics.
The arXiv team has been expanding the publicly available
submission statistics. This is a tableau-based application
with tabs at the top for various displays of submission
stats and filters on the side bar to drill down by topic.
(Hover over the charts to view individual categories.) The
data is meant to be displayed on a monthly basis withdownload options.

CHAPTER 1 PREVIEW

41

Artificial IntelligenceIndex Report 2021

N E S TA

APPENDIX

CHAPTER 1:
RESEARCH &D E V E LO P M E N T

G I T H U B S TA R S

Prepared by Joel Kliger and Juan Mateos-Garcia

SourceSource

GitHub: star-history (available at star history website)
was used to retrieve the data.

Details can be found in the following publication:
Deep Learning, Deep Change? Mapping the Developmentof the Artificial Intelligence General Purpose Technology

Methodology
Deep learning papers were identified through a topic
modeling analysis of the abstracts of arXiv papers in
the CS (computer science) and stats.ML (statistics:
machine learning category) arXiv categories. The data
was enriched with institutional affiliation and geographic
information from the Microsoft Academic Graph andthe Global Research Identifier. Nesta’s arXlive tool isavailable here.

Access the CodeThe code for data collection and processing can be found
here; or, without the infrastructure overhead here.

CHAPTER 1 PREVIEW

Methodology
The visual in the report shows the number of stars forvarious GitHub repositories over time. The repositories
include the following:
apache/incubator-mxnet, BVLC/cafe, cafe2/cafe2, dmlc/
mxnet, fchollet/keras, Microsoft/CNTK, pytorch/pytorch,scikit-learn/scikit-learn, tensorflow/tensorflow, Theano/
Theano, Torch/Torch7.

NuanceThe GitHub Archive currently does not provide a way
to count when users remove a star from a repository.
Therefore, the reported data slightly overestimates thenumber of stars. A comparison with the actual numberof stars for the repositories on GitHub reveals that the
numbers are fairly close and that the trends remainunchanged.

42