Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 1:

Research andDevelopment

Chapter 1 Preview

1Artificial IntelligenceIndex Report 2023

CHAPTER 1 PREVIEW:

Research and Development
Overview

3

Computer Vision27

Chapter Highlights

4

Natural Language Processing

28

Speech Recognition29

1.1 Publications

5

Overview

5

Total Number of AI Publications

5

1.2 Trends in Significant
Machine Learning Systems

By Type of Publication

6

General Machine Learning Systems

By Field of Study

7

System Types

30

By Sector

8

Sector Analysis

31

Cross-Country Collaboration

10

National Affiliation

32

Cross-Sector Collaboration

12

		Systems

32

13

		Authorship

34

AI Journal Publications

30
30

Overview

13

Parameter Trends

35

By Region

14

Compute Trends

37

By Geographic Area

15

Citations16

National Affiliation

39

17

Parameter Count

41

Overview

17

Training Compute42

By Region

18

Training Cost

43

By Geographic Area

19

Citations20

1.3 AI Conferences

45

AI Repositories

21

Conference Attendance

45

Overview

21

By Region

22

1.4 Open-Source AI Software

47

By Geographic Area

23

Projects

47

Citations

24

Stars

49

Appendix

50

AI Conference Publications

Narrative Highlight:
Top Publishing Institutions
All Fields

Chapter 1 Preview

25
25

Large Language and Multimodal Models

39

ACCESS THE PUBLIC DATA

2

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development

Overview
This chapter captures trends in AI R&D. It begins by examining AI publications,
including journal articles, conference papers, and repositories. Next it considers dataon significant machine learning systems, including large language and multimodal
models. Finally, the chapter concludes by looking at AI conference attendance andopen-source AI research. Although the United States and China continue to dominate
AI R&D, research efforts are becoming increasingly geographically dispersed.

Chapter 1 Preview

3Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development

Chapter Highlights
The United States and China
had the greatest number ofcross-country collaborations in AI
publications from 2010 to 2021,
although the pace of collaboration
has since slowed.
The number of AI research collaborations between
the United States and China increased roughly 4times since 2010, and was 2.5 times greater than thecollaboration totals of the next nearest country pair,
the United Kingdom and China. However, the total
number of U.S.-China collaborations only increasedby 2.1% from 2020 to 2021, the smallest year-overyear growth rate since 2010.

AI research is on the rise, acrossthe board. The total number of AI publications
has more than doubled since 2010. The specific AI
topics that continue to dominate research include
pattern recognition, machine learning,and computer vision.

China continues to lead in total
AI journal, conference, andrepository publications.
The United States is still ahead in terms of AI
conference and repository citations, but those leads
are slowly eroding. Still, the majority of the world’s
large language and multimodal models (54% in 2022)
are produced by American institutions.

Chapter 1 Preview

Industry races aheadof academia.
Until 2014, most significant machinelearning models were released byacademia. Since then, industry has taken
over. In 2022, there were 32 significant
industry-produced machine learningmodels compared to just three producedby academia. Building state-of-the-art
AI systems increasingly requires large
amounts of data, computer power, andmoney—resources that industry actors
inherently possess in greater amounts
compared to nonprofits and academia.

Large language models
are getting bigger andmore expensive.
GPT-2, released in 2019, consideredby many to be the first large language
model, had 1.5 billion parameters andcost an estimated $50,000 USD totrain. PaLM, one of the flagship large
language models launched in 2022,
had 540 billion parameters and cost an
estimated $8 million USD—PaLM wasaround 360 times larger than GPT-2 andcost 160 times more. It’s not just PaLM:
Across the board, large language andmultimodal models are becoming larger
and pricier.

4Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

This section draws on data from the Center for Security and Emerging Technology (CSET) at Georgetown University. CSET maintains amerged corpus of scholarly literature that includes Digital Science’s Dimensions, Clarivate’s Web of Science, Microsoft Academic Graph,
China National Knowledge Infrastructure, arXiv, and Papers With Code. In that corpus, CSET applied a classifier to identify Englishlanguage publications related to the development or application of AI and ML since 2010. For this year’s report, CSET also used selectChinese AI keywords to identify Chinese-language AI papers; CSET did not deploy this method for previous iterations of the AI Index report.1
In last year’s edition of the report, publication trends were reported up to the year 2021. However, given that there is a significant lag in thecollection of publication metadata, and that in some cases it takes until the middle of any given year to fully capture the previous year’s
publications, in this year’s report, the AI Index team elected to examine publication trends only through 2021, which we, along with CSET,
are confident yields a more fully representative report.

1.1 Publications
Overview

publication and citation data by region for AI journal
articles, conference papers, repositories, and patents.

The figures below capture the total number
of English-language and Chinese-language AI

Total Number of AI Publications

publications globally from 2010 to 2021—by type,Figure 1.1.1 shows the number of AI publications inaffiliation, cross-country collaboration, and cross-the world. From 2010 to 2021, the total number ofindustry collaboration. The section also breaks down

AI publications more than doubled, growing from
200,000 in 2010 to almost 500,000 in 2021.

Number of AI Publications in the World, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report496.01

Number of AI Publications (in Thousands)

500

400

300

200

100

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.1

1 See the Appendix for more information on CSET’s methodology. For more on the challenge of defining AI and correctly capturing relevant bibliometric data, see the AI Index team’s
discussion in the paper “Measurement in AI Policy: Opportunities and Challenges.”

Chapter 1 Preview

5Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Type of Publication

book chapters, theses, and unknown document types

Figure 1.1.2 shows the types of AI publications released

made up the remaining 10% of publications. While

globally over time. In 2021, 60% of all published AI

journal and repository publications have grown 3

documents were journal articles, 17% were conference

and 26.6 times, respectively, in the past 12 years, thepapers, and 13% were repository submissions. Books,

number of conference papers has declined since 2019.

Number of AI Publications by Type, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report300

293.48, Journal

Number of AI Publications (in Thousands)

270
240
210
180
150
120
90

85.09, Conference
65.21, Repository60

29.88, Thesis
13.77, Book Chapter
5.82, Unknown
2.76, Book

30
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.2

Chapter 1 Preview

6Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Field of Studyroughly doubled while the number of machine learning

Figure 1.1.3 shows that publications in patternpapers has roughly quadrupled. Following those tworecognition and machine learning have experienced

topic areas, in 2021, the next most published AI fields

the sharpest growth in the last half decade. Sinceof study were computer vision (30,075), algorithm

2015, the number of pattern recognition papers has(21,527), and data mining (19,181).

Number of AI Publications by Field of Study (Excluding Other AI), 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Publications (in Thousands)

60

59.36, Pattern Recognition50

42.55, Machine Learning
40

30

30.07, Computer Vision20

21.53, Algorithm
19.18, Data Mining

10

14.99, Natural Language Processing
11.57, Control Theory
10.37, Human–Computer Interaction
6.74, Linguistics

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.3

Chapter 1 Preview

7Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Sector
This section shows the number of AI publications

1.1.5).2 The education sector dominates in each region.

affiliated with education, government, industry,

The level of industry participation is highest in the

nonprofit, and other sectors—first globally (Figure

United States, then in the European Union. Since

1.1.4), then looking at the United States, China, and2010, the share of education AI publications has been

the European Union plus the United Kingdom (Figure

dropping in each region.

AI Publications (% of Total) by Sector, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report80%
75.23%, Education
70%

AI Publications (% of Total)

60%

50%

40%

30%

20%
13.60%, Nonpro t
10%

7.21%, Industry
3.74%, Government
0.22%, Other

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.4

2 The categorization is adapted based on the Global Research Identifier Database (GRID). Healthcare, including hospitals and facilities, is included under nonprofit. Publications affiliated withstate-sponsored universities are included in the education sector.

Chapter 1 Preview

8Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Publications (% of Total) by Sector and Geographic Area, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

69.17%
Education

69.23%
77.85%
14.82%

Nonpro t

18.63%
11.73%
12.60%

Industry

7.90%
5.47%
3.21%

Government

3.92%
4.74%
United States

0.20%

European Union and United Kingdom

Other 0.33%

China

0.20%
0%

Chapter 1 Preview

10%

20%

30%

40%
50%
AI Publications (% of Total)

60%

70%

80%
Figure 1.1.5

9

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Country Collaboration
Cross-border collaborations between academics,

By far, the greatest number of collaborations in the

researchers, industry experts, and others are a key

past 12 years took place between the United Statescomponent of modern STEM (science, technology,

and China, increasing roughly four times since 2010.

engineering, and mathematics) development that

However the total number of U.S.-China collaborations

accelerate the dissemination of new ideas and theonly increased by 2.1% from 2020 to 2021, the smallest

growth of research teams. Figures 1.1.6 and 1.1.7 depict

year-over-year growth rate since 2010.

the top cross-country AI collaborations from 2010
to 2021. CSET counted cross-country collaborations
as distinct pairs of countries across authors for eachpublication (e.g., four U.S. and four Chinese-affiliatedauthors on a single publication are counted as one
U.S.-China collaboration; two publications between

The next largest set of collaborations was between
the United Kingdom and both China and the UnitedStates. In 2021, the number of collaborations between
the United States and China was 2.5 times greaterthan between the United Kingdom and China.

the same authors count as two collaborations).

United States and China Collaborations in AI Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

10.47

Number of AI Publications (in Thousands)

10

8

6

4

2

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.6

Chapter 1 Preview

10Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Country Collaborations in AI Publications (Excluding U.S. and China), 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

4.13, United Kingdom and China
4.04, United States and United Kingdom

Number of AI Publications (in Thousands)

4

3.42, United States and Germany
32.80, China and Australia
2.61, United States and Australia

2
1.83, United States and France

1

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.7

Chapter 1 Preview

11Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Sector Collaboration
The increase in AI research outside of academia has

educational institutions (12,856); and educational

broadened and grown collaboration across sectors

and government institutions (8,913). Collaborations

in general. Figure 1.1.8 shows that in 2021 educational

between educational institutions and industry haveinstitutions and nonprofits (32,551) had the greatest

been among the fastest growing, increasing 4.2 times

number of collaborations; followed by industry andsince 2010.

Cross-Sector Collaborations in AI Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

32.55, Education and Nonpro t

Number of AI Publications (in Thousands)

30

25

20

15
12.86, Industry and Education
10

8.91, Education and Government

52.95, Government and Nonpro t
2.26, Industry and Nonpro t
0.63, Industry and Government

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.8

Chapter 1 Preview

12Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Journal Publications
Overview
After growing only slightly from 2010 to 2015, the number of AI journal publications grew around 2.3 times since2015. From 2020 to 2021, they increased 14.8% (Figure 1.1.9).

Number of AI Journal Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

293.48

Number of AI Journal Publications (in Thousands)

300

250

200

150

100

50

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.9

Chapter 1 Preview

13Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region3
Figure 1.1.10 shows the share of AI journal publications

East Asia and the Pacific; Europe and Central Asia;

by region between 2010 and 2021. In 2021, East Asia

as well as North America have been declining.

and the Pacific led with 47.1%, followed by Europe

During that period, there has been an increase in

and Central Asia (17.2%), and then North America

publications from other regions such as South Asia;(11.6%). Since 2019, the share of publications fromand the Middle East and North Africa.

AI Journal Publications (% of World Total) by Region, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

50%

AI Journal Publications (% of World Total)

47.14%, East Asia and Paci c

40%

30%

20%
17.20%, Europe and Central Asia
11.61%, North America
6.93%, Unknown
6.75%, South Asia
4.64%, Middle East and North Africa
2.66%, Latin America and the Caribbean
2.30%, Rest of the World
0.77%, Sub-Saharan Africa

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.10

3 Regions in this chapter are classified according to the World Bank analytical grouping.

Chapter 1 Preview

14Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area4
Figure 1.1.11 breaks down the share of AI journal

throughout, with 39.8% in 2021, followed by the

publications over the past 12 years by geographic

European Union and the United Kingdom (15.1%),

area. This year’s AI Index included India in recognition

then the United States (10.0%). The share of Indianof the increasingly important role it plays in the

publications has been steadily increasing—from 1.3%

AI ecosystem. China has remained the leader

in 2010 to 5.6% in 2021.

AI Journal Publications (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Journal Publications (% of World Total)

40%

39.78%, China

30%

22.70%, Rest of the World20%
15.05%, European Union and United Kingdom
10.03%, United States

10%

6.88%, Unknown
5.56%, India

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.11

4 In this chapter we use “geographic area” based on CSET’s classifications, which are disaggregated not only by country, but also by territory. Further, we count the European Union and theUnited Kingdom as a single geographic area to reflect the regions’ strong history of research collaboration.

Chapter 1 Preview

15Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
China’s share of citations in AI journal publications

1.1.12). China, the European Union and the United

has gradually increased since 2010, while those of theKingdom, and the United States accounted for 65.7%

European Union and the United Kingdom, as well asof the total citations in the world.

those of the United States, have decreased (Figure

AI Journal Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%

29.07%, China

AI Journal Citations (% of World Total)

27.37%, Rest of the World25%
21.51%, European Union and United Kingdom
20%

15.08%, United States

15%

10%

6.05%, India

5%

0.92%, Unknown

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.12

Chapter 1 Preview

16Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Conference Publications
Overview
The number of AI conference publications peaked in 2019, and fell 20.4% below the peak in 2021 (Figure 1.1.13).
The total number of 2021 AI conference publications, 85,094, was marginally greater than the 2010 total of 75,592.

Number of AI Conference Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Conference Publications (in Thousands)

100
85.09
80

60

40

20

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.13

Chapter 1 Preview

17Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region
Figure 1.1.14 shows the number of AI conference

East Asia and the Pacific continues to rise, accounting

publications by region. As with the trend in journal

for 36.7% in 2021, followed by Europe and Central

publications, East Asia and the Pacific; Europe

Asia (22.7%), and then North America (19.6%). The

and Central Asia; and North America account forpercentage of AI conference publications in South Asia

the world’s highest numbers of AI conference

saw a noticeable rise in the past 12 years, growing frompublications. Specifically, the share represented by

3.6% in 2010 to 8.5% in 2021.

AI Conference Publications (% of World Total) by Region, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report40%

AI Conference Publications (% of World Total)

36.72%, East Asia and Paci c
35%
30%
25%
22.66%, Europe and Central Asia
20%

19.56%, North America

15%
10%

8.45%, South Asia
3.82%, Middle East and North Africa
3.07%, Latin America and the Caribbean
2.76%, Unknown
2.35%, Rest of the World
0.60%, Sub-Saharan Africa

5%
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.14

Chapter 1 Preview

18Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area
In 2021, China produced the greatest share of thecame in third at 17.2% (Figure 1.1.15). Mirroring trends

world’s AI conference publications at 26.2%, having

seen in other parts of the research and development

overtaken the European Union and the United

section, India’s share of AI conference publications is

Kingdom in 2017. The European Union plus the United

also increasing.

Kingdom followed at 20.3%, and the United States

AI Conference Publications (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Conference Publications (% of World Total)

30%
26.84%, Rest of the World
26.15%, China

25%

20.29%, European Union and United Kingdom

20%

17.23%, United States
15%

10%
6.79%, India
5%
2.70%, Unknown
0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.15

Chapter 1 Preview

19Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
Despite China producing the most AI conference

conference citations, with 23.9%, followed by China’s

publications in 2021, Figure 1.1.16 shows that

22.0%. However, the gap between American andthe United States had the greatest share of AI

Chinese AI conference citations is narrowing.

AI Conference Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

35%

AI Conference Citations (% of World Total)

30%

25.57%, Rest of the World
23.86%, United States
22.02%, China
21.59%, European Union and United Kingdom

25%

20%

15%

10%
6.09%, India

5%0%0.87%, Unknown
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.16

Chapter 1 Preview

20Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Repositories
Overview
Publishing pre-peer-reviewed papers on repositories

share their findings before submitting them to journals

of electronic preprints (such as arXiv and SSRN)

and conferences, thereby accelerating the cycle ofhas become a popular way for AI researchers to

information discovery. The number of AI repository

disseminate their work outside traditional avenues for

publications grew almost 27 times in the past 12 years

publication. These repositories allow researchers to(Figure 1.1.17).

Number of AI Repository Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Repository Publications (in Thousands)

65.21
60

50

40

30

20

10

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.17

Chapter 1 Preview

21Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region
Figure 1.1.18 shows that North America has

by East Asia and the Pacific has grown significantly

maintained a steady lead in the world share of AI

since 2010 and continued growing from 2020 torepository publications since 2016. Since 2011, the2021, a period in which the year-over-year share ofshare of repository publications from Europe andNorth American as well European and Central AsianCentral Asia has declined. The share represented

repository publications declined.

AI Repository Publications (% of World Total) by Region, 2010–21

AI Repository Publications (% of World Total)

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%
26.32%, North America
23.99%, Unknown
21.40%, Europe and Central Asia

20%

17.88%, East Asia and Paci c

10%
3.41%, South Asia
3.06%, Middle East and North Africa
1.81%, Rest of the World
1.80%, Latin America and the Caribbean
0.34%, Sub-Saharan Africa

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.18

Chapter 1 Preview

22Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area
While the United States has held the lead in the(Figure 1.1.19). In 2021, the United States accounted

percentage of global AI repository publications sincefor 23.5% of the world’s AI repository publications,2016, China is catching up, while the European Union

followed by the European Union plus the United

plus the United Kingdom’s share continues to drop

Kingdom (20.5%), and then China (11.9%).

AI Repository Publications (% of World Total) by Geographic Area, 2010–21

AI Repository Publications (% of World Total)

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%

23.48%, United States
23.18%, Unknown
20.54%, European Union and United Kingdom

20%

18.07%, Rest of the World

11.87%, China
10%

2.85%, India
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.19

Chapter 1 Preview

23Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
In the citations of AI repository publications, Figure

a dominant lead over the European Union plus the

1.1.20 shows that in 2021 the United States topped

United Kingdom (21.5%), as well as China (21.0%).

the list with 29.2% of overall citations, maintaining

AI Repository Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Repository Citations (% of World Total)

40%

30%

29.22%, United States

21.79%, Rest of the World
21.52%, European Union and United Kingdom
20.98%, China

20%

10%

4.59%, Unknown
1.91%, India
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.20

Chapter 1 Preview

24Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions
All Fields

University, the University of the Chinese Academy

Since 2010, the institution producing the greatestof Sciences, Shanghai Jiao Tong University,

number of total AI papers has been the Chineseand Zhejiang University.5 The total number of

Academy of Sciences (Figure 1.1.21). The next

publications released by each of these institutions

top four are all Chinese universities: Tsinghua

in 2021 is displayed in Figure 1.1.22.

Top Ten Institutions in the World in 2021 Ranked by Number of AI Publications in All Fields, 2010–21

Rank

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

1

1, Chinese Academy of Sciences2

2, Tsinghua University

3

3, University of Chinese Academy of Sciences

44, Shanghai Jiao Tong University

55, Zhejiang University

6

6, Harbin Institute of Technology

77, Beihang University

88, University of Electronic Science and Technology of China

99, Peking University

10

10, Massachusetts Institute of Technology
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

Figure 1.1.21

5 It is important to note that many Chinese research institutions are large, centralized organizations with thousands of researchers. It is therefore not entirely surprising that,
purely by the metric of publication count, they outpublish most non-Chinese institutions.

Chapter 1 Preview

25Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)
Top Ten Institutions in the World by Number of AI Publications in All Fields, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

5,099

Tsinghua University

3,373

University of Chinese Academyof Sciences

2,904

Shanghai Jiao Tong University

2,703

Zhejiang University

2,590

Harbin Institute of Technology

2,016

Beihang University

1,970

University of Electronic Scienceand Technology of China

1,951

Peking University

1,893

Massachusetts Institute ofTechnology

1,745
0

500

1,000

1,500

2,000
2,500
3,000
Number of AI Publications

3,500

4,000

4,500

5,000

Figure 1.1.22

Chapter 1 Preview

26Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)Computer Vision
In 2021, the top 10 institutions publishing the greatest number of AI computer vision publications were
all Chinese (Figure 1.1.23). The Chinese Academy of Sciences published the largest number of such
publications, with a total of 562.

Top Ten Institutions in the World by Number of AI Publications in Computer Vision, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

562

Shanghai Jiao Tong University

316

University of Chinese Academyof Sciences

314

Tsinghua University

296

Zhejiang University

289

Beihang University

247

Wuhan University

231

Beijing Institute of Technology

229

Harbin Institute of Technology

210

Tianjin University
0

182
100

200

300
Number of AI Publications

400

500

Figure 1.1.23

Chapter 1 Preview

27Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)Natural Language Processing
American institutions are represented to a

took second place (140 publications), followed bygreater degree in the share of top NLP publishers

Microsoft (134). In addition, 2021 was the first year

(Figure 1.1.24). Although the Chinese Academy ofAmazon and Alibaba were represented among theSciences was again the world’s leading institution

top-ten largest publishing NLP institutions.

in 2021 (182 publications), Carnegie Mellon

Top Ten Institutions in the World by Number of AI Publications in Natural Language Processing, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

182

Carnegie Mellon University

140

Microsoft (United States)

134

Tsinghua University

127

Carnegie Mellon University
Australia

116

Google (United States)

116

Peking University

113

University of Chinese Academyof Sciences

112

Alibaba Group (China)

100

Amazon (United States)

98
0

10

20

30

40

50

60

70

80
90
100
110
120
Number of AI Publications

130

140

150

160

170

180

190

Figure 1.1.24

Chapter 1 Preview

28Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)
Speech Recognition
In 2021, the greatest number of speech recognition papers came from the Chinese Academy of Sciences(107), followed by Microsoft (98) and Google (75) (Figure 1.1.25). The Chinese Academy of Sciences
reclaimed the top spot in 2021 from Microsoft, which held first position in 2020.

Top Ten Institutions in the World by Number of AI Publications in Speech Recognition, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

107

Microsoft (United States)

98

Google (United States)

75

University of Chinese Academyof Sciences

66

Tsinghua University

61

University of Scienceand Technology of China

59

Carnegie Mellon University

57

Tencent (China)

57

Chinese University of Hong Kong

55

Amazon (United States)

54

0

10

20

30

40

50
60
70
Number of AI Publications

80

90

100

110

Figure 1.1.25

Chapter 1 Preview

29Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Epoch AI is a collective of researchers investigating and forecasting the development of advanced AI. Epoch curates a database ofsignificant AI and machine learning systems that have been released since the 1950s. There are different criteria under which the
Epoch team decides to include particular AI systems in their database; for example, the system may have registered a state-of-the-art
improvement, been deemed to have been historically significant, or been highly cited.
This subsection uses the Epoch database to track trends in significant AI and machine learning systems. The latter half of the chapter
includes research done by the AI Index team that reports trends in large language and multimodal models, which are models trained onlarge amounts of data and adaptable to a variety of downstream applications.

1.2 Trends in Significant
Machine Learning Systems
General Machine
Learning Systems

System Types

The figures below report trends among all machine

was language (Figure 1.2.1). There were 23 significant

Among the significant AI machine learning systems
released in 2022, the most common class of system

learning systems included in the Epoch dataset. For
reference, these systems are referred to as significant
machine learning systems throughout the subsection.

AI language systems released in 2022, roughly six
times the number of the next most common system
type, multimodal systems.

Number of Significant Machine Learning Systems by Domain, 2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

23

Multimodal

4

Drawing3

Vision2

Speech2

Text-to-Video

1

Other

1

Games

1

0

2

4

6

8
10
12
14
16
Number of Signi cant Machine Learning Systems

18

20

22

24
Figure 1.2.16

6 There were 38 total significant AI machine learning systems released in 2022, according to Epoch; however, one of the systems, BaGuaLu, did not have a domain classificationand is therefore omitted from Figure 1.2.1.

Chapter 1 Preview

30Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Sector Analysis
Which sector among industry, academia, or nonprofit

machine learning systems compared to just three

has released the greatest number of significant

produced by academia. Producing state-of-the-art

machine learning systems? Until 2014, most machine

AI systems increasingly requires large amounts oflearning systems were released by academia.

data, computing power, and money; resources that

Since then, industry has taken over (Figure 1.2.2). In

industry actors possess in greater amounts compared2022, there were 32 significant industry-produced

to nonprofits and academia.

Number of Significant Machine Learning Systems by Sector, 2002–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Number of Signi cant Machine Learning Systems

35
32, Industry
30

25

20

15

10

5

3, Academia
2, Research Collective
1, Industry-Academia Collaboration
0, Nonpro t

0
2002

2004

Chapter 1 Preview

2006

2008

2010

2012

2014

2016

2018

2020

2022

Figure 1.2.2

31

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

National Affiliation
In order to paint a picture of AI’s evolving

or AI-research firm, was headquartered. In 2022,

geopolitical landscape, the AI Index research

the United States produced the greatest number

team identified the nationality of the authors whoof significant machine learning systems with 16,

contributed to the development of each significant

followed by the United Kingdom (8) and China (3).

machine learning system in the Epoch dataset.

Moreover, since 2002 the United States has outpaced

7

the United Kingdom and the European Union, as well

Systems

as China, in terms of the total number of significant

Figure 1.2.3 showcases the total number ofmachine learning systems produced (Figure 1.2.4).

significant machine learning systems attributed to

Figure 1.2.5 displays the total number of significant

researchers from particular countries.8 A researcher

machine learning systems produced by country since

is considered to have belonged to the country in2002 for the entire world.

Number of Significant Machine Learning Systems byCountry, 2022
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

United States

16

United Kingdom

8

China

3

Canada2Germany

2

France

1

India

1

Israel1

Russia1

Singapore

1

30
25
20
16, United States

15

12, European Union andUnited Kingdom

1053, China
0
2022

2020

2018

2016

2014

2012

2010

2008

2006

2004

2
4
6
8
10
12
14
16
Number of Signi cant Machine Learning Systems

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

2002

0

Number of Significant Machine Learning Systems by
Select Geographic Area, 2002–22
Number of Significant Machine Learning Systems

which their institution, for example a university

Figure 1.2.3

Figure 1.2.4

7 The methodology by which the AI Index identified authors’ nationality is outlined in greater detail in the Appendix.
8 A machine learning system is considered to be affiliated with a particular country if at least one author involved in creating the model was affiliated with that country.
Consequently, in cases where a system has authors from multiple countries, double counting may occur.

Chapter 1 Preview

32Artificial IntelligenceIndex Report 2023

Number of

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Machine Learning Systems by Country, 2002–22 (Sum)

Source: AI Index, 2022 | Chart: 2023 AI Index Report

0
1–10
11–20
21–60
61–255Figure 1.2.5

Chapter 1 Preview

33Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Authorship
Figures 1.2.6 to 1.2.8 look at the total number ofin 2022 the United States had the greatest number ofauthors, disaggregated by national affiliation, that

authors producing significant machine learning systems,

contributed to the launch of significant machinewith 285, more than double that of the United Kingdom

learning systems. As was the case with total systems,

and nearly six times that of China (Figure 1.2.6).

Number of Authors of Significant Machine Learning
Systems by Country, 2022

Number of Authors of Significant Machine Learning
Systems by Select Geographic Area, 2002–22

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

United States

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

400

285

United Kingdom

350

139

China

300

Canada21

Israel13

Sweden8Germany

7

Number of Authors

49

200
155, European Union andUnited Kingdom

150
100

Russia 3

50

India 249, China

0France 1

2022

2020

2018

2016

2014

2012

Figure 1.2.6

2010

300

2008

Source: AI Index, 2022 | Chart: 2023 AI Index Report

250

2006

Number of Authors of

100
150
200
Number of Authors

2004

50

2002

0

285, United States

250

Figure 1.2.7

Machine Learning Systems by Country, 2002–22 (Sum)

0
1–10
11–20
21–60
61–180
181–370
371–680
681–2000

Chapter 1 Preview

Figure 1.2.8

34

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Parameter Trends
Parameters are numerical values that are learned bydataset by sector. Over time, there has been a steadymachine learning models during training. The value ofincrease in the number of parameters, an increase thatparameters in machine learning models determines

has become particularly sharp since the early 2010s.

how a model might interpret input data and makeThe fact that AI systems are rapidly increasing theirpredictions. Adjusting parameters is an essential

parameters is reflective of the increased complexity ofstep in ensuring that the performance of a machine

the tasks they are being asked to perform, the greater

learning system is optimized.

availability of data, advancements in underlying
hardware, and most importantly, the demonstrated

Figure 1.2.9 highlights the number of parameters of

performance of larger models.

the machine learning systems included in the Epoch

Number of Parameters of Significant Machine Learning Systems by Sector, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

1.0e+14
Academia

Industry

Industry-Academia Collaboration

Nonpro t

Research Collective

Number of Parameters (Log Scale)

1.0e+121.0e+101.0e+81.0e+6

1.0e+41.0e+2

1950

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002 2006

2010

2014

2018

2022
Figure 1.2.9

Chapter 1 Preview

35Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Figure 1.2.10 demonstrates the parameters of machine learning systems by domain. In recent years, there has
been a rise in parameter-rich systems.

Number of Parameters of Significant Machine Learning Systems by Domain, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

VisionGames

1.0e+12

Number of Parameters (Log Scale)

1.0e+101.0e+81.0e+6

1.0e+41.0e+2

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002

2006

2010

2014

2018

2022
Figure 1.2.10

Chapter 1 Preview

36Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Compute Trends

machine learning systems has increased exponentially

The computational power, or “compute,” of AI

in the last half-decade (Figure 1.2.11).9 The growing

systems refers to the amount of computational

demand for compute in AI carries several importantresources needed to train and run a machine

implications. For example, more compute-intensive

learning system. Typically, the more complex amodels tend to have greater environmental impacts,system is, and the larger the dataset on which it istrained, the greater the amount of compute required.
The amount of compute used by significant AI

and industrial players tend to have easier accessto computational resources than others, such asuniversities.

Training Compute (FLOP) of Significant Machine Learning Systems by Sector, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Academia

Industry

Industry-Academia Collaboration

Nonpro t

Research Collective

1.0e+24

Training Compute (FLOP – Log Scale)

1.0e+211.0e+18

1.0e+15

1.0e+121.0e+91.0e+6

1.0e+31.0e+0

1950

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002 2006

2010

2014

2018

2022
Figure 1.2.11

9 FLOP stands for “Floating Point Operations” and is a measure of the performance of a computational device.

Chapter 1 Preview

37Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Since 2010, it has increasingly been the case that of all machine learning systems, language models aredemanding the most computational resources.

Training Compute (FLOP) of Significant Machine Learning Systems by Domain, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

VisionGames

1.0e+24

Training Compute (FLOP – Log Scale)

1.0e+211.0e+18

1.0e+15

1.0e+121.0e+91.0e+6

1.0e+3
1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002

2006

2010

2014

2018

2022
Figure 1.2.12

Chapter 1 Preview

38Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Large Language andMultimodal Models

are starting to be widely deployed in the real world.

Large language and multimodal models, sometimes

national affiliation of the authors responsible for

National Affiliation

called foundation models, are an emerging andincreasingly popular type of AI model that is trainedon huge amounts of data and adaptable to a variety
of downstream applications. Large language andmultimodal models like ChatGPT, DALL-E 2, and MakeA-Video have demonstrated impressive capabilities andThis year the AI Index conducted an analysis of thereleasing new large language and multimodal models.10
The majority of these researchers were from American
institutions (54.2%) (Figure 1.2.13). In 2022, for the firsttime, researchers from Canada, Germany, and India
contributed to the development of large language andmultimodal models.

Authors of Select Large Language and Multimodal Models (% of Total) by Country, 2019–22

Authors of Large Language and Multimodal Models (% of Total)

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

100%

80%

60%
54.02%, United States40%

21.88%, United Kingdom
8.04%, China
6.25%, Canada
5.80%, Israel
3.12%, Germany
0.89%, India
0.00%, Korea

20%

0%
2019

2020

2021

2022
Figure 1.2.13

Figure 1.2.14 offers a timeline view of the large

PaLM (540B). The only Chinese large language andlanguage and multimodal models that have been

multimodal model released in 2022 was GLM-130B,

released since GPT-2, along with the national

an impressive bilingual (English and Chinese) model

affiliations of the researchers who produced thecreated by researchers at Tsinghua University. BLOOM,

models. Some of the notable American large

also launched in late 2022, was listed as indeterminate

language and multimodal models released ingiven that it was the result of a collaboration of more

2022 included OpenAI’s DALL-E 2 and Google’s

than 1,000 international researchers.

10 The AI models that were considered to be large language and multimodal models were hand-selected by the AI Index steering committee. It is possible that this selection may have omitted
certain models.

Chapter 1 Preview

39Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Timelineand National Affiliation of Select Large Language and Multimodal Model Releases
Source: AI Index, 2022 | Chart: 2023 AI Index Report2023-Jan

BLOOM

2022-Oct

GLM-130B

2022-Jul

Minerva (540B)

2022-Apr

Jurassic-X
Stable Diffusion (LDM-KL-8-G) PaLM (540B)
GPT-NeoX-20B

2022-Jan

AlphaCode
Gopher2021-Oct

Megatron-Turing NLG 530B

2021-Jul

Codex
HyperClova Wu Dao 2.0

2021-Apr

InstructGPTImagenOPT-175B
DALL·E 2
Chinchilla

PanGu-alpha

CogView

Wu Dao - Wen Yuan

Jurassic-1-Jumbo
ERNIE 3.0
GPT-J-6B
GPT-Neo
DALL-E

2021-Jan
2020-OctERNIE-GEN (large)

2020-Jul

GPT-3 175B (davinci)

2020-Apr

Turing NLG

Meena

2020-Jan
2019-Oct

Megatron-LM (Original, 8.3B)

2019-Jul

T5-11B

T5-3B

Grover-Mega

United States
United Kingdom
China
United States,
United Kingdom,
Germany, India
Korea

CanadaIsraelGermany
Indeterminate

2019-Apr

GPT-2
2019-Jan
Figure 1.2.1411

11 While we were conducting the analysis to produce Figure 1.2.14, Irene Solaiman published a paper that has a similar analysis. We were not aware of the paper at the time of our research.

Chapter 1 Preview

40Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Parameter CountOver time, the number of parameters of newly released

Google in 2022, had 540 billion, nearly 360 times

large language and multimodal models has massively

more than GPT-2. The median number of parameters

increased. For example, GPT-2, which was the firstin large language and multimodal models is increasinglarge language and multimodal model released in 2019,

exponentially over time (Figure 1.2.15).

only had 1.5 billion parameters. PaLM, launched byNumber of Parameters of Select Large Language and Multimodal Models, 2019–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

3.2e+12

Wu Dao 2.0

Number of Parameters (Log Scale)

1.0e+12

Megatron-Turing NLG 530B

3.2e+11HyperClova

GPT-3 175B (davinci)

Gopher PaLM (540B)
Jurassic-1-Jumbo

PanGu-αMinerva (540B)

OPT-175B

1.0e+11BLOOM

GLM-130B
Chinchilla

3.2e+10

GPT-NeoX-20B

Turing NLG

DALL-E

T5-11BGPT-J-6B ERNIE 3.0

1.0e+103.2e+91.0e+9

Codex

Megatron-LM (Original, 8.3B)
T5-3B Meena

Jurassic-X
DALL·E 2

CogView
Wu Dao - Wen Yuan
Stable Di usion (LDM-KL-8-G)
GPT-Neo

GPT-2
Grover-Mega
ERNIE-GEN (large)

3.2e+8

ov

-N

22

20

ug
-A
22
20
un
-J y
22 a
20 2-M r
2 p
20 2-A ar2
20 2-M b
2 e
2022-F
20
ec
-D
21
20
ct
-O
21
20
ug
-A
21 ul
20 1-J n
2 u
20 1-J ay
2
20 1-M r
2 p
20 1-A ar
2
20 -M
21
20
an
-J
21

20

ug
-A
20
20
ay
-M
20
20
eb
-F
20 an
20 0-J
2
20

ct
-O
19 ep
20 9-S
1
20

ay
-M

19
20

eb
-F

19
20

Figure 1.2.15

Chapter 1 Preview

41Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Training Compute
The training compute of large language and multimodal

reasoning problems, was roughly nine times greatermodels has also steadily increased (Figure 1.2.16). The

than that used for OpenAI’s GPT-3, which wascompute used to train Minerva (540B), a large language

released in June 2022, and roughly 1839 times greaterand multimodal model released by Google in Junethan that used for GPT-2 (released February 2019).

2022 that displayed impressive abilities on quantitative

Training Compute (FLOP) of Select Large Language and Multimodal Models, 2019–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

3.2e+24

Training Compute (FLOP – Log Scale)

1.0e+24

GPT-3 175B (davinci)

3.2e+23

Meena

1.0e+23

T5-11B

3.2e+22T5-3B

1.0e+223.2e+21

PaLM (540B)
Minerva (540B)
Megatron-Turing NLG 530B
OPT-175B
GopherJurassic-1-Jumbo
Chinchilla
BLOOM
GPT-NeoX-20B
PanGu-αDALL-E
Stable Diffusion
HyperClova

Turing NLG

Megatron-LM (Original, 8.3B)
GPT-2

CogView

AlphaCodeGLM-130B

GPT-J-6B
GPT-Neo
Wu Dao - Wen Yuan

1.0e+213.2e+20
1.0e+203.2e+19
1.0e+19

ERNIE 3.0

3.2e+18
1.0e+18

ov

-N

22

20

ug
-A
22
20
un
-J y
22 a
20 -M
22 pr
20 2-A ar2
20 2-M b
2 e
2022-F
20
ec
-D
21
20
ct
-O
21
20
ug
-A
21 ul
20 1-J
2
20
ay
-M
21 pr
20 1-A ar
2
20 -M
21
20
an
-J
21

20

ay
-M
20
20
eb
-F
20 an
20 0-J
2
20

ct
-O
19 ep
20 9-S
1
20

eb
-F

19
20

Figure 1.2.16

Chapter 1 Preview

42Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Training Cost
A particular theme of the discourse around large

estimate with the tag of mid, high, or low: mid wherelanguage and multimodal models has to do with their

the estimate is thought to be a mid-level estimate,hypothesized costs. Although AI companies rarely speakhigh where it is thought to be an overestimate, andopenly about training costs, it is widely speculated that

low where it is thought to be an underestimate. In

these models cost millions of dollars to train and willcertain cases, there was not enough data to estimate

become increasingly expensive with scale.

the training cost of particular large language andmultimodal models, therefore these models wereThis subsection presents novel analysis in which theomitted from our analysis.

AI Index research team generated estimates for the
training costs of various large language and multimodal

The AI Index estimates validate popular claims thatmodels (Figure 1.2.17). These estimates are based on thelarge language and multimodal models are increasingly

hardware and training time disclosed by the models’costing millions of dollars to train. For example,

authors. In cases where training time was not disclosed,

Chinchilla, a large language model launched by

we calculated from hardware speed, training compute,

DeepMind in May 2022, is estimated to have cost $2.1

and hardware utilization efficiency. Given the possible

million, while BLOOM’s training is thought to have costvariability of the estimates, we have qualified each

$2.3 million.

Estimated Training Cost of Select Large Language and Multimodal Models
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Mid

High

Low

11.35

10

8.55

8.01

8
6
4

2020

BLOOM

0.16
GLM-130B

OPT-175B

Stable Di usion (LDM-KL-8-G)

PaLM (540B)

2.29
1.03
Minerva (540B)

1.69
0.60

Chinchilla

Codex

0.24
GPT-NeoX-20B

ERNIE 3.0

2021

0.09

AlphaCode

0.14
Gopher

0.01

Megatron-Turing NLG 530B

0.27
HyperClova

GPT-J-6B

GPT-Neo

GPT-3 175B

Turing NLG

Meena

2019

Wu Dao - Wen Yuan

0.23 0.02 0.09 0.43

0.11

0.05

T5-11B

0

2.11

1.80

1.47

DALL-E

1.97

2

GPT-2

Training Cost
(in Millions of U.S. Dollars)

12

2022
Figure 1.2.17

12 See Appendix for the complete methodology behind the cost estimates.

Chapter 1 Preview

43Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

There is also a clear relationship between the cost of large language and multimodal models and their size.
As evidenced in Figures 1.2.18 and 1.2.19, the large language and multimodal models with a greater number ofparameters and that train using larger amounts of compute tend to be more expensive.

Estimated Training Cost of Select Large Language
and Multimodal Models and Number of Parameters
Source: AI Index, 2022 | Chart: 2023 AI Index Report

HyperClova
GLM-130B

1.0e+11

OPT-175B
BLOOM
GPT-3 175B
Chinchilla

AlphaCode5.0e+10

GPT-NeoX-20B
Turing NLG

2.0e+10Codex

ERNIE 3.0

T5-11B

DALL-EGPT-J-6B

5.0e+9Wu Dao - Wen Yuan

2.0e+9GPT-2

GPT-Neo

Meena

PaLM (540B)
Megatron-Turing NLG 530B
OPT-175B
GopherChinchilla

1.0e+24
Training Compute (FLOP – Log Scale)

Number of Parameters (Log Scale)

Minerva (540B)

Megatron-Turing NLG 530B
Gopher

2.0e+111.0e+10Source: AI Index, 2022 | Chart: 2023 AI Index Report

PaLM (540B)

Minerva (540B)

5.0e+11

Estimated Training Cost of Select Large Language andMultimodal Models and Training Compute (FLOP)

GPT-NeoX-20B
AlphaCodeDALL-E

Turing NLG

1.0e+22

Meena

GLM-130B Stable Diffusion

BLOOM
T5-11BGPT-J-6B
GPT-Neo

GPT-2
Wu Dao - Wen Yuan

1.0e+20

Stable Di usion

ERNIE 3.0

1.0e+18

1.0e+9
10k

100k
1M
10M
Training Cost (in U.S. Dollars - Log Scale)Figure 1.2.18

Chapter 1 Preview

10k

100k
1M
10M
Training Cost (in U.S. Dollars - Log Scale)Figure 1.2.19

44

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.3 AI Conferences

AI conferences are key venues for researchers to share their work and connect with peers and collaborators. Conference attendance is anindication of broader industrial and academic interest in a scientific field. In the past 20 years, AI conferences have grown in size, number,and prestige. This section presents data on the trends in attendance at major AI conferences.

1.3 AI Conferences
Conference AttendanceInternational Conference on Principles of Knowledge

After a period of increasing attendance, the total

strictly in-person.

Representation and Reasoning (KR) were both held

attendance at the conferences for which the AI

Neural Information Processing Systems (NeurIPS)

Index collected data dipped in 2021 and again incontinued to be one of the most attended

2022 (Figure 1.3.1).13 This decline may be attributed

conferences, with around 15,530 attendees (Figure

to the fact that many conferences returned to hybrid

1.3.2).14 The conference with the greatest one-or in-person formats after being fully virtual inyear increase in attendance was the International

2020 and 2021. For example, the International Joint

Conference on Robotics and Automation (ICRA),

Conference on Artificial Intelligence (IJCAI) and thefrom 1,000 in 2021 to 8,008 in 2022.

Number of Attendees at Select AI Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

90

Number of Attendees (in Thousands)

80
70
60

59.45

50
40
30
20
10
0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.1

13 This data should be interpreted with caution given that many conferences in the last few years have had virtual or hybrid formats. Conference organizers report thatmeasuring the exact attendance numbers at virtual conferences is difficult, as virtual conferences allow for higher attendance of researchers from around the world.
14 In 2021, 9,560 of the attendees attended NeurIPS in-person and 5,970 remotely.

Chapter 1 Preview

45Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.3 AI Conferences

Attendance at Large Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

30

Number of Attendees (in Thousands)

25

20

15.53, NeurIPS15

10.17, CVPR
8.01, ICRA
7.73, ICML

10

5.35, ICLR
4.32, IROS
3.56, AAAI

5

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.2

Attendance at Small Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Number of Attendees (in Thousands)

3.50

3.00

2.50

2.01, IJCAI

2.00

1.50
1.09, FaccT

1.00

0.66, UAI
0.50, AAMAS
0.39, ICAPS

0.50

0.12, KR

0.00
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.3

Chapter 1 Preview

46Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

GitHub is a web-based platform where individuals and coding teams can host, review, and collaborate on various code repositories.
GitHub is used extensively by software developers to manage and share code, collaborate on various projects, and support open-source
software. This subsection uses data provided by GitHub and the OECD.AI policy observatory. These trends can serve as a proxy for someof the broader trends occuring in the world of open-source AI software not captured by academic publication data.

1.4 Open-Source AI Software
Projects
A GitHub project is a collection of files thatsoftware project. Since 2011, the total number of

can include the source code, documentation,

AI-related GitHub projects has steadily increased,

configuration files, and images that constitute a

growing from 1,536 in 2011 to 347,934 in 2022.

Number of GitHub AI Projects, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

350

348

Number of AI Projects (in Thousands)

300

250

200

150

100

50

0
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.1

Chapter 1 Preview

47Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

As of 2022, a large proportion of GitHub AI projects

United Kingdom (17.3%), and then the United States

were contributed by software developers in India(14.0%). The share of American GitHub AI projects(24.2%) (Figure 1.4.2). The next most represented

has been declining steadily since 2016.

geographic area was the European Union and the

GitHub AI Projects (% Total) by Geographic Area, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

42.11%, Rest of the World
40%
35%

AI Projects (% of Total)

30%
25%

24.19%, India

20%
17.30%, European Union and United Kingdom
15%

14.00%, United States

10%5%2.40%, China
0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.2

Chapter 1 Preview

48Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

Stars
GitHub users can bookmark or save a repository

Figure 1.4.3 shows the cumulative number of starsof interest by “starring” it. A GitHub star is similar

attributed to projects belonging to owners of various

to a “like” on a social media platform and indicatesgeographic areas. As of 2022, GitHub AI projects

support for a particular open-source project. Some offrom the United States received the most stars,

the most starred GitHub repositories include libraries

followed by the European Union and the United

like TensorFlow, OpenCV, Keras, and PyTorch, which

Kingdom, and then China. In many geographic areas,

are widely used by software developers in the AI

the total number of new GitHub stars has leveled off

coding community.

in the last few years.

Number of GitHub Stars by Geographic Area, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

Number of Cumulative GitHub Stars (in Millions)3.50

3.44, United States

3.00
2.69, Rest of the World
2.50
2.34, European Union and United Kingdom
2.00

1.53, China

1.50

1.00

0.50

0.46, India

0.00
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.3

Chapter 1 Preview

49Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

AppendixCenter for Security andEmerging Technology,
Georgetown University

Methodology

Prepared by Sara Abdulla and James Dunham

publications. To identify AI publications, CSET used an

The Center for Security and Emerging Technology

since 2010 that appear AI-relevant.4 CSET researchers(CSET) is a policy research organization within
Georgetown University’s Walsh School of Foreign
Service that produces data-driven research at theintersection of security and technology, providing
nonpartisan analysis to the policy community.
For more information about how CSET analyzes
bibliometric and patent data, see the Country Activity
Tracker (CAT) documentation on the Emerging
Technology Observatory’s website.1 Using CAT, users
can also interact with country bibliometric, patent,
and investment data.2

To create the merged corpus, CSET deduplicatedacross the listed sources using publication metadata,
and then combined the metadata for linked
English-language subset of this corpus: publications
developed a classifier for identifying AI-related
publications by leveraging the arXiv repository, where
authors and editors tag papers by subject. Additionally,
CSET uses select Chinese AI keywords to identify
Chinese-language AI papers.5
To provide a publication’s field of study, CSET
matches each publication in the analytic corpus
with predictions from Microsoft Academic Graph’s
field-of-study model, which yields hierarchical labels
describing the published research field(s) of study andcorresponding scores.6 CSET researchers identified

Publications from CSET Merged Corpus of
Scholarly Literature

the most common fields of study in our corpus of

Sourcepublications in all other fields as “Other AI.” English-

CSET’s merged corpus of scholarly literature

language AI-relevant publications were then tallied by

combines distinct publications from Digital Science’s

their top-scoring field and publication year.

Dimensions, Clarivate’s Web of Science, Microsoft
Academic Graph, China National Knowledge
Infrastructure, arXiv, and Papers With Code.3

AI-relevant publications since 2010 and recorded

CSET also provided year-by-year citations for AIrelevant work associated with each country. A
publication is associated with a country if it has at

1 https://eto.tech/tool-docs/cat/
2 https://cat.eto.tech/3 All CNKI content is furnished by East View Information Services, Minneapolis, Minnesota, USA.
4 For more information, see James Dunham, Jennifer Melot, and Dewey Murdick, “Identifying the Development and Application of Artificial Intelligence in Scientific Text,” arXiv [cs.DL],
May 28, 2020, https://arxiv.org/abs/2002.07143.
5 This method was not used in CSET’s data analysis for the 2022 HAI Index report.
6 These scores are based on cosine similarities between field-of-study and paper embeddings. See Zhihong Shen, Hao Ma, and Kuansan Wang, “A Web-Scale System for Scientific
Knowledge Exploration,” arXiv [cs.CL], May 30, 2018, https://arxiv.org/abs/1805.12216.

Chapter 1 Preview

50Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

available for all publications; those without counts

Epoch National
Affiliation Analysis

weren’t included in the citation analysis. Over 70% of

The AI forecasting research group Epoch maintains

English-language AI papers published between 2010

a dataset of landmark AI and ML models, along withand 2020 have citation data available.

accompanying information about their creators andleast one author whose organizational affiliation(s)are located in that country. Citation counts aren’t

CSET counted cross-country collaborations as
distinct pairs of countries across authors for eachpublication. Collaborations are only counted once:

publications, such as the list of their (co)authors, numberof citations, type of AI task accomplished, and amountof compute used in training.

For example, if a publication has two authors fromThe nationalities of the authors of these papers havethe United States and two authors from China, it isimportant implications for geopolitical AI forecasting.

counted as a single United States-China collaboration.

As various research institutions and technology

Additionally, publication counts by year and by
publication type (e.g., academic journal articles,
conference papers) were provided where available.
These publication types were disaggregated byaffiliation country as described above.
CSET also provided publication affiliation sector(s)where, as in the country attribution analysis, sectors
were associated with publications through authors’affiliations. Not all affiliations were characterized interms of sectors; CSET researchers relied primarily

companies start producing advanced ML models, theglobal distribution of future AI development may shift
or concentrate in certain places, which in turn affects
the geopolitical landscape because AI is expected to
become a crucial component of economic and military
power in the near future.
To track the distribution of AI research contributions on
landmark publications by country, the Epoch dataset iscoded according to the following methodology:
1. A snapshot of the dataset was taken on

on GRID from Digital Science for this purpose, andNovember 14, 2022. This includes papers about

not all organizations can be found in or linked tolandmark models, selected using the inclusion

GRID. Where the affiliation sector is available, papers

criteria of importance, relevance, and uniqueness,were counted toward these sectors, by year. Cross-as described in the Compute Trends dataset

sector collaborations on academic publications

documentation.8

7were calculated using the same method as in the
cross-country collaborations analysis. We use HAI’s
standard regions mapping for geographic analysis,
and the same principles for double-counting apply forregions as they do for countries.

2. The authors are attributed to countries basedon their affiliation credited on the paper. Forinternational organizations, authors are attributed
to the country where the organization is
headquartered, unless a more specific location
is indicated. The number of authors from eachcountry represented are added up and recorded.

7 See https://www.grid.ac/ for more information about the GRID dataset from Digital Science.
8 https://epochai.org/blog/compute-trends; see note on “milestone systems.”

Chapter 1 Preview

51Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

If an author has multiple affiliations in different

ImagenOPT-175B

countries, they are split between those

InstructGPTPaLM (540B)

countries proportionately.

Jurassic-1-Jumbo

PanGu-alphaJurassic-X

Stable Diffusion (LDM-

Meena

KL-8-G)

Megatron-LM (original,

T5-3B

8.3B)

T5-11B

Megatron-Turing NLG

Turing NLG

530B

Wu Dao 2.0

Minerva (540B)

Wu Dao – Wen Yuan

9

3. E ach paper in the dataset is normalized toequal value by dividing the counts on each
paper from each country by the total number ofauthors on that paper.10
4. All of the landmark publications are aggregatedwithin time periods (e.g., monthly or yearly) with
the normalized national contributions added up
to determine what each country’s contribution
to landmark AI research was during each time
period.
5. The contributions of different countries arecompared over time to identify any trends.

Large Language andMultimodal Models Training
Cost Analysis
Cost estimates for the models were based directly

Large Language andMultimodal Models

on the hardware and training time if these were

The following models were identified by members

training compute, and hardware utilization efficiency.11of the AI Index Steering Committee as the large

Training time was then multiplied by the closest cost

language and multimodal models that would berate for the hardware the AI Index could find for the

included as part of the large language and multimodalorganization that trained the model. If price quotes

model analysis:

were available before and after the model’s training,

AlphaCodeGLM-130B

BLOOM

GopherChinchilla

GPT-2

Codex

GPT-3 175B (davinci)

CogView

GPT-J-6B

DALL-E

GPT-Neo

DALL-E 2

GPT-NeoX-20B

ERNIE 3.0

Grover-Mega

ERNIE-GEN (large)

HyperCLOVA

disclosed by the authors; otherwise, the AI Index
calculated training time from the hardware speed,

the AI Index interpolated the hardware’s cost rate
along an exponential decay curve.
The AI Index classified training cost estimates as
high, middle, or low. The AI Index called an estimate
high if it was an upper bound or if the true cost wasmore likely to be lower than higher: For example,
PaLM was trained on TPU v4 chips, and the AI Index
estimated the cost to train the model on these chips
from Google’s public cloud compute prices, but the

9 For example, an author employed by both a Chinese university and a Canadian technology firm would be counted as 0.5 researchers from China and 0.5 from Canada.
10 This choice is arbitrary. Other plausible alternatives include weighting papers by their number of citations, or assigning greater weight to papers with more authors.
11 Hardware utilization rates: Every paper that reported the hardware utilization efficiency during training provided values between 30% and 50%. The AI Index used the reported numbers
when available, or used 40% when values were not provided.

Chapter 1 Preview

52Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

internal cost to Google is probably lower than what

each file’s revision history. The analysis of GitHub

they charge others to rent their hardware. The AI

data could shed light on relevant metrics about who

Index called an estimate low if it was a lower boundis developing AI software, where, and how fast, andor if the true cost was likely higher: For example,

who is using which development tools. These metrics

ERNIE was trained on NVIDIA Tesla v100 chips andcould serve as proxies for broader trends in the field

published in July 2021; the chips cost $0.55 per hourof software development and innovation.

in January 2023, so the AI Index could get a low

Identifying AI Projects

estimate of the cost using this rate, but the training
hardware was probably more expensive two years
earlier. Middle estimates are a best guess, or thosethat equally well might be lower or higher.

AI Conferences
The AI Index reached out to the organizers of various
AI conferences in 2022 and asked them to provide
information on total attendance. Some conferences
posted their attendance totals online; when this was
the case, the AI Index used those reported totals anddid not reach out to the conference organizers.

GitHubThe GitHub data was provided to the AI Index
through OECD.AI, an organization with whom
GitHub partners that provides data on opensource AI software. The AI Index reproduces themethodological note that is included by OECD.AI on
its website, for the GitHub Data.

Background
Since its creation in 2007, GitHub has become
the main provider of internet hosting for softwaredevelopment and version control. Many technology
organizations and software developers use GitHub
as a primary place for collaboration. To enable
collaboration, GitHub is structured into projects, or“repositories,” which contain a project’s files andChapter 1 Preview

Arguably, a significant portion of AI softwaredevelopment takes place on GitHub. OECD.AI
partners with GitHub to identify public AI projects—
or “repositories”—following the methodology
developed by Gonzalez et al.,2020. Using the 439
topic labels identified by Gonzalez et al.—as well as
the topics “machine learning,” “deep learning,” and“artificial intelligence”—GitHub provides OECD.
AI with a list of public projects containing AI code.
GitHub updates the list of public AI projects on aquarterly basis, which allows OECD.AI to capture
trends in AI software development over time.

Obtaining AI Projects’ Metadata
OECD.AI uses GitHub’s list of public AI projects
to query GitHub’s public API and obtain more
information about these projects. Project metadata
may include the individual or organization thatcreated the project; the programming language(s)(e.g., Python) and development tool(s) (e.g., Jupyter
Notebooks) used in the project; as well as information
about the contributions—or “commits”—made to it,which include the commit’s author and a timestamp.
In practical terms, a contribution or “commit” is anindividual change to a file or set of files. Additionally,
GitHub automatically suggests topical tags to eachproject based on its content. These topical tags need
to be confirmed or modified by the project owner(s)to appear in the metadata.

53

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

Mapping Contributions to AI Projects to a
Country

As of October 2021, 71.2% of the contributions toContributions to public AI projects are mapped

this methodology. However, a decreasing trend in

to a country based on location information at thethe share of AI projects for which a location can becontributor level and at the project level.

identified is observed in time, indicating a possible laga) Location information at the contributor level:• GitHub’s “Location” field: Contributors canprovide their location in their GitHub account.
Given that GitHub’s location field accepts freetext, the location provided by contributors is

public AI projects were mapped to a country using

in location reporting.

Measuring Contributions to AI Projects
Collaboration on a given public AI project is measured
by the number of contributions—or “commits”—madeto it.

not standardized and could belong to different

To obtain a fractional count of contributions bylevels (e.g., suburban, urban, regional, or

country, an AI project is divided equally by the total

national). To allow cross-country comparisons,number of contributions made to it. A country’s total

Mapbox is used to standardize all available

contributions to AI projects is therefore given by thelocations to the country level.

sum of its contributions—in fractional counts—to each• Top level domain: Where the location fieldis empty or the location is not recognized, acontributor’s location is assigned based on his
or her email domain (e.g., .fr, .us, etc.).
b) Location information at the project level:• Project information: Where no location

AI project. In relative terms, the share of contributions
to public AI projects made by a given country is theratio of that country’s contributions to each of the
AI projects in which it participates over the total
contributions to AI projects from all countries.
In future iterations, OECD.AI plans to include
additional measures of contribution to AI software

information is available at the contributor

development, such as issues raised, comments, andlevel, information at the repository or project

pull requests.

level is exploited. In particular, contributions

Identifying Programming Languages andDevelopment Tools Used in AI Projects

from contributors with no location information
to projects created or owned by a known

GitHub uses file extensions contained in a project toorganization are automatically assigned theautomatically tag it with one or more programmingorganization’s country (i.e., the country wherelanguages and/or development tools. This implies thatits headquarters are located). For example,

more than one programming language or development

contributions from a contributor with notool could be used in a given AI project.

location information to an AI project owned byMicrosoft will be assigned to the United States.
If the above fails, a contributor’s location field is left
blank.

Chapter 1 Preview

54Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

Measuring the Quality of AI Projects
Two quality measures are used to classify public AI
projects:• Project impact: The impact of an AI project isgiven by the number of managed copies (i.e.,“forks”) made of that project.
• Project popularity: The impact of an AI projectis given by the number of followers (i.e., “stars”)
received by that project.
Filtering by project impact or popularity could help
identify countries that contribute the most to highquality projects.

Measuring Collaboration
Two countries are said to collaborate on a specificpublic AI software development project if there is
at least one contributor from each country with atleast one contribution (i.e., “commit”) to the project.
Domestic collaboration occurs when two contributors
from the same country contribute to a project.

Chapter 1 Preview

55