Artificial IntelligenceIndex Report 2023

CHAPTER 2:

Technical
Performance

Artificial IntelligenceIndex Report 2023

CHAPTER 2 PREVIEW:

Technical Performance
Overview

4

Chapter Highlights

5

Narrative Highlight: A Closer Look at
Progress in Image Generation
22
Visual Reasoning

24

2.1 What’s New in 2022: A Timeline

6

Visual Question Answering (VQA)
Challenge

2.2 Computer Vision—Image

13

Narrative Highlight: The Rise of Capable
Multimodal Reasoning Systems
25

Image Classification

13

ImageNet
Face Detection and Recognition
National Institute of Standards andTechnology Face Recognition
Vendor Test (FRVT)
Deepfake Detection
Celeb-DF
Human Pose Estimation
MPII
Semantic Segmentation
Cityscapes Challenge, Pixel-Level
Semantic Labeling Task
Medical Image Segmentation

14

Activity Recognition

28

16

Narrative Highlight: A Closer Lookat the Progress of Video Generation

16
17
17
18
18
19
20

Chapter 2 Preview

28

Kinetics-400, Kinetics-600, Kinetics-700 28

Object Detection

CIFAR-10 and STL-10

2.3 Computer Vision—Video

15

19

Image Generation

Visual Commonsense Reasoning (VCR) 27

13

Kvasir-SEG
Common Objects in Context (COCO)

24

20
21
21

30

2.4 Language

31

English Language Understanding

31

SuperGLUE

31

Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)

32

Narrative Highlight: Just How Much
Better Have Language Models Become?

34

Narrative Highlight: Planning andReasoning in Large Language Models

35

Text Summarization
arXiv and PubMed

36
36

2

Artificial IntelligenceIndex Report 2023

CHAPTER 2 PREVIEW (CONT’D):

Technical Performance
Natural Language Inference
Abductive Natural Language
Inference (aNLI)
Sentiment Analysis
SST-5 Fine-Grained Classification

2.7 Hardware

47

MLPerf Training Time

47

37

MLPerf Inference49

38

Trends in GPUs

50

37

38

Multitask Language Understanding39

2.8 Environment

52

Massive Multitask Language
Understanding (MMLU)

39

Environmental Impact ofSelect Large Language Models

52

Machine Translation (MT)
Number of Commercially Available
MT Systems

4041

Speech Recognition

41

Narrative Highlight:
Whisper

54

40

2.5 Speech
VoxCeleb

Narrative Highlight: Using AI to
Optimize Energy Usage

41
42

2.6 Reinforcement Learning

44

Reinforcement Learning Environments

44

Procgen44

Narrative Highlight:
Benchmark Saturation

46

2.9 AI for Science

55

Accelerating Fusion Science ThroughLearned Plasma Control

55

Discovering Novel Algorithms for
Matrix Manipulation With AlphaTensor

55

Designing Arithmetic Circuits With
Deep Reinforcement Learning

56

Unlocking de Novo Antibody Design
With Generative AI

56

Appendix

57

ACCESS THE PUBLIC DATA

Chapter 2 Preview

3Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance

Overview
This year’s technical performance chapter features analysis of the technical progress in
AI during 2022. Building on previous reports, this chapter chronicles advancement incomputer vision, language, speech, reinforcement learning, and hardware. Moreover,this year this chapter features an analysis on the environmental impact of AI, a discussionof the ways in which AI has furthered scientific progress, and a timeline-style overview
of some of the most significant recent AI developments.

Chapter 2 Preview

4Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance

Chapter Highlights
Performance saturation on
traditional benchmarks.

Generative AI breaks into
the public consciousness.

AI continued to post state-of-the-art results,
but year-over-year improvement on many
benchmarks continues to be marginal.
Moreover, the speed at which benchmark
saturation is being reached is increasing.
However, new, more comprehensive
benchmarking suites such as BIG-bench andHELM are being released.

2022 saw the release of text-to-image models
like DALL-E 2 and Stable Diffusion, text-tovideo systems like Make-A-Video, and chatbots
like ChatGPT. Still, these systems can beprone to hallucination, confidently outputting
incoherent or untrue responses, making it hard
to rely on them for critical applications.

AI systems become
more flexible.
Traditionally AI systems have performed wellon narrow tasks but have struggled acrossbroader tasks. Recently released models
challenge that trend; BEiT-3, PaLI, andGato, among others, are single AI systems
increasingly capable of navigating multiple
tasks (for example, vision, language).

AI is both helping and harming
the environment.
New research suggests that AI systems canhave serious environmental impacts. According
to Luccioni et al., 2022, BLOOM’s training run
emitted 25 times more carbon than a single air
traveler on a one-way trip from New York to
San Francisco. Still, new reinforcement learningmodels like BCOOLER show that AI systems
can be used to optimize energy usage.

Chapter 2 Preview

Capable language models
still struggle with reasoning.
Language models continued to improvetheir generative capabilities, but new
research suggests that they still struggle
with complex planning tasks.

The world’s best new scientist… AI? AI models are starting to rapidly
accelerate scientific progress and in 2022
were used to aid hydrogen fusion, improvethe efficiency of matrix manipulation, andgenerate new antibodies.

AI starts to build better AI.
Nvidia used an AI reinforcement learning agent
to improve the design of the chips that power AI
systems. Similarly, Google recently used one ofits language models, PaLM, to suggest ways toimprove the very same model. Self-improving AI
learning will accelerate AI progress.

5Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

The technical performance chapter begins with an overview of some of the most significant technical developments in AI during 2022,
as selected by the AI Index Steering Committee.

2.1 What’s New in 2022: A Timeline
Feb. 2, 2022

DeepMind Releases AlphaCode
AlphaCode, an AI system that writes computer programs
at a competitive level, achieves a rank within the top 54%
of participants in a human programming competition. This
represents an improvement on the more complex problemsolving tasks with which AI has traditionally struggled.
Figure 2.1.1

Feb. 16, 2022

DeepMind Trains Reinforcement Learning Agent toControl Nuclear Fusion Plasma in a Tokamak
Nuclear fusion is a potential source of clean, limitlessenergy, but producing such energy in tokamaks is difficult
due to a lack of experimental data. DeepMind simulated
optimal tokamak management, an example of how AI canaccelerate science and combat climate change.

March 10, 2022

Figure 2.1.2

IndicNLG Benchmarks Natural Language Generation for Indic Languages
An international research collective launches IndicNLG, a collection of datasets for benchmarking
natural language generation for 11 Indic languages. The creation of IndicNLG increases thepotential for AI systems to generate language in more diverse, non-English linguistic settings.

Figure 2.1.3

Chapter 2 Preview

6Artificial IntelligenceIndex Report 2023

March 24, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Meta AI Releases Make-A-Scene
Make-A-Scene is a text-to-image AI model thatenables users to generate images through text.
Make-A-Scene is one of many text-to-image
models released in 2022.

Figure 2.1.4

April 5, 2022

Google Releases PaLM
Google’s AI team trains one of the world’s
largest language models, PaLM. Made upof 540 billion parameters, PaLM reinforces
the belief that researchers can improve
performance on large language models bysimply training them on more data.

April 13, 2022

Figure 2.1.5

OpenAI Releases DALL-E 2
DALL-E 2, a text-to-image AI system that can create
realistic art and images from textual descriptions, isreleased to the public, igniting a generative AI craze.

Figure 2.1.6

May 12, 2022

DeepMind Launches Gato
Gato is a new reinforcement learning agent
capable of doing a wide range of tasks suchas robotic manipulation, game playing, image
captioning, and natural language generation.
The release of such models suggests that AI
systems are becoming better at generalization.

Figure 2.1.7

Chapter 2 Preview

7Artificial IntelligenceIndex Report 2023

May 23, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Google Releases Imagen
Imagen is a text-to-image diffusion model capableof producing images with a high degree ofphotorealism. Imagen’s launch also comes withthe release of DrawBench, a challenging new
benchmark for text-to-image systems.
Figure 2.1.8

June 9, 2022

442 Authors Across 132 Institutions Team Up to Launch BIG-bench
In order to better challenge increasingly capable large language models, a team of 442 authors
across 132 institutions launch the Beyond the Imitation Game benchmark (BIG-bench). The
benchmark consists of 204 tasks ranging from linguistics, childhood development, math,common-sense reasoning, biology, physics, social bias, and software development.

Figure 2.1.9

June 21, 2022

GitHub Makes Copilot Available asa Subscription-Based Service for
Individual Developers
Copilot is a generative AI system capableof turning natural language prompts
into coding suggestions across multiple
languages. Similar systems include OpenAI’s
Codex and Salesforce’s CodeGen. Surveys
suggest that Copilot makes coders moreproductive and less frustrated.
Figure 2.1.10

Chapter 2 Preview

8Artificial IntelligenceIndex Report 2023

July 8, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Nvidia Uses Reinforcement Learning to
Design Better-Performing GPUs
Nvidia uses its AI systems to improve theperformance of its latest H100 class of GPU chips.
GPUs being essential to AI training, this is one
example of how AI is starting to develop better AI.

Figure 2.1.11

July 11, 2022

Meta Announces‘No Language Left Behind’
No Language Left Behind (NLLB) isa family of models that can translate
across 200 distinct languages. NLLB is
one of the first systems that can perform
well across a wide range of low-resource
languages like Kamba and Lao.
Figure 2.1.12

Aug 4, 2022

Tsinghua Researchers Launch GLM-130B
Chinese researchers affiliated with Tsinghua
University release GLM-130B, a large language
model that outperforms others such as Meta’s
OPT, Hugging Face’s BLOOM, and OpenAI’s
original GPT-3.

Aug 22, 2022

Figure 2.1.13

Stability AI Releases Stable Diffusion
Stable Diffusion is an open-source text-to-image
diffusion-based model, meaning users can freely use
the model weights to generate their own images. Stable
Diffusion is trained on existing images created by humans
and gives no credit or acknowledgment, leaving open
questions around the ethical use of image generators.
Figure 2.1.14

Chapter 2 Preview

9Artificial IntelligenceIndex Report 2023

Sept 21, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

OpenAI Launches Whisper
Whisper is a large-scale speech-recognition systemtrained on roughly 700,000 hours of audio data andcapable of respectable performance on various speech
recognition tasks. The fact that Whisper required neither
supervised pre-training nor unsupervised training with
fine-tuning yet was able to achieve strong performanceby merely increasing training data further validates theapproach of increasingly scaling AI models.

Sept 29, 2022

Figure 2.1.15

Meta Releases Make-A-Video
Make-A-Video is a system that allows users
to create videos from short text descriptions.
The quality of the videos is high and again
demonstrates the validity of the scaling
approach.

Figure 2.1.16

Oct 5, 2022

DeepMind Launches AlphaTensor
AlphaTensor is an AI reinforcement-learningbased system able to discover new andefficient algorithms for matrix manipulation.
Matrix manipulation is essential to a wide
range of digital practices and is a processthat researchers have been trying to makemore efficient for decades.

Figure 2.1.17

Chapter 2 Preview

10Artificial IntelligenceIndex Report 2023

Oct 20, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Google Uses PaLM to Improve
the Reasoning of PaLM
Google researchers use one of
their existing language models,
PaLM, to improve the reasoningof the very same model. This
process is yet another exampleof AI systems using their own

Figure 2.1.18

knowledge to improve.

Nov 9, 2022

International ResearchGroup Releases BLOOM
A collaboration of over 100
researchers from across theglobe develop an open-accesslanguage model called BLOOM.
BLOOM impresses with its
public release and for furthering

Figure 2.1.19

the possibilities of international
collaboration in AI research.

Nov 16, 2022

Stanford Researchers Release HELM
As part of an attempt to judge new language models according to more unified standards, Stanford
researchers develop a new benchmarking approach for large language models called Holistic Evaluationof Language Models (HELM). The launch of HELM is evidence of the AI community’s attempt to developtransparency around increasingly powerful, capable, and influential large language models.

Figure 2.1.20

Chapter 2 Preview

11Artificial IntelligenceIndex Report 2023

Nov 22, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Meta Releases CICERO
CICERO is the first AI to play inthe top 10% of human participantsin the game Diplomacy. CICERO’s
launch shows that AI systems have
improved in strategic reasoning, adomain in which they have traditionallystruggled, and are capable ofeffectively convincing humans to go
along with their objectives.

Nov 30, 2022

Figure 2.1.21

OpenAI Launches ChatGPT
ChatGPT is an impressive,
publicly usable chatbot capableof writing university-level
essays. Months after launching,
ChatGPT reaches 100 million
monthly active users, making it
the fastest-growing consumer
application in history. ChatGPT’s
release caps a year in which
generative AI became a partof the zeitgeist, and raises

Figure 2.1.22

questions about the effect that
AI will have on the future ofhumanity.

Chapter 2 Preview

12Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Computer vision is the subfield of AI that teaches machines to understand images and videos. Computer vision technologies have avariety of important real-world applications, such as autonomous driving, crowd surveillance, sports analytics, and video-game creation.
This section tracks progress in computer vision across several different task domains which include: (1) image classification, (2)face detection and recognition, (3) deepfake detection, (4) human pose estimation, (5) semantic segmentation, (6) medical image
segmentation, (7) object detection, (8) image generation, and (9) visual reasoning.

2.2 Computer Vision—Image
Image Classification
Image classification is the ability of machines to

A Demonstration of Image Classification
Source: Krizhevsky et al., 2012

categorize objects in images (Figure 2.2.1).

ImageNet
ImageNet is one of the most widely usedbenchmarks for image classification. This dataset
includes over 14 million images across 20,000
different object categories such as “strawberry” or“balloon.” Performance on ImageNet is measured
through various accuracy metrics. Top-1 accuracy
measures the degree to which the top prediction
generated by an image classification model for agiven image actually matches the image’s label.
As of 2022, the best image classification system on
ImageNet has a top-1 accuracy rate of 91.0% (Figure

Figure 2.2.1

2.2.2). Although the current image classification
capabilities of state-of-the-art systems is 27.7
percentage points better than a decade ago, lastyear saw a very marginal 0.1 percentage point
improvement in classification accuracy.

Chapter 2 Preview

13Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

ImageNet Challenge: Top-1 Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

91.00%, With Extra Training Data

90%88.50%, Without Extra Training Data

Top-1 Accuracy (%)

85%

80%

75%

70%

65%

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.2

Face Detection and
Recognition

A Demonstration of Face Detection and Recognition
Source: Forbes, 2020

Facial detection and recognition is the ability of AI
systems to identify faces or individuals in images
or videos (Figure 2.2.3). Currently, many facialrecognition systems are able to successfully identify
close to 100% of faces, even on challenging datasets(Figure 2.2.4).

Figure 2.2.3

Chapter 2 Preview

14Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT):
Veri cation Accuracy by Dataset
Source: National Institute of Standards and Technology, 2022 | Chart: 2023 AI Index Report

1.0000

False Non-Match Rate: FMNR (Log-Scale)

0.50000.20000.10000.05000.0297, WILD Photos @ FMR = 1e-5
0.02000.01000.00500.0032, BORDER Photos @ FMR = 1e-6
0.0021, MUGSHOT Photos @ FMR = 1e-50.0019, MUGSHOT Photos ≥ 12 YRS @ FMR = 1e-50.0016, VISABORDER Photos @ FMR = 1e-6

0.00200.00100.0006, VISA Photos @ FMR = 1e-6

0.0005
2017

2018

2019

2020

2021

2022
Figure 2.2.4

National Institute of Standards andTechnology Face Recognition Vendor Test
(FRVT)

others. Facial detection capacity is measured by the

Progress on facial recognition can be tracked

match the face in an image to that of a person.

through the National Institute of Standards andTechnology’s Face Recognition Vendor Test. This
test tracks how well different facial recognition
algorithms perform on various homeland security
tasks, such as identification of child trafficking

false non-match rate (FNMR), otherwise known aserror rate, which is the rate at which a model fails to

As of 2022, the top-performing models on all of the
FRVT datasets, with the exception of WILD Photos,
each posted an error rate below 1%, and as low as a
0.06% error rate on the VISA Photos dataset.

victims and cross-verification of visa images, among

Chapter 2 Preview

15Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Deepfake Detection

Real-Life Deepfake: President Zelenskyy Calling
for the Surrender of Ukrainian Soldiers

The ability of AI systems to create synthetic images

Source: NPR, 2022

that are sometimes indistinguishable from real ones
has led to the creation of deepfakes, images orvideos that appear to be real but are actually fake. In
the last year, there was a widely circulated deepfake
video of Ukrainian president Volodymyr Zelenskyy
surrendering (Figure 2.2.5).

Celeb-DFCeleb-DF is presently one of the most challenging
Figure 2.2.5

deepfake detection benchmarks. This dataset
is composed of 590 original celebrity YouTube

algorithm on Celeb-DF came from researchers atvideos that have been manipulated into thousands

Deakin University in Australia. Their JDFD modelof deepfakes. This year’s top deepfake detection

posted an AUC score of 78 (Figure 2.2.6).

Celeb-DF: Area Under Curve Score (AUC)
Source: arXiv, 2022 | Chart: 2023 AI Index Report

78.00

Area Under Curve Score (AUC)

75

70

65
2018

2019

2020

2021

2022
Figure 2.2.6

Chapter 2 Preview

16Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Human Pose Estimation

A Demonstration of Human Pose Estimation
Source: Cong et al., 2022

Human pose estimation is the task ofestimating the position of the human bodyfrom images (Figure 2.2.7).

MPII
MPII is a dataset of over 25,000 annotatedimages which contains annotations ofmore than 40,000 people doing 410 humanactivities. On MPII, this year’s top model,
Figure 2.2.7

ViTPose, correctly estimated 94.3% of
keypoints (human joints), which represented
a small 0.2 percentage point increase fromthe previous state-of-the-art result posted in2020 (Figure 2.2.8).

MPII: Percentage of Correct Keypoints (PCK)
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

95%

Percentage of Correct Keypoints (PCK)

94.30%

90%

85%

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.8

Chapter 2 Preview

17Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Semantic Segmentation

A Demonstration of Semantic Segmentation
Source: Cityscapes Dataset, 2022

Semantic segmentation involves assigning individual
image pixels to specific categories (for example,
human, bicycle, or street) (Figure 2.2.9).

Cityscapes Challenge,
Pixel-Level Semantic Labeling Task
The Cityscapes dataset is used to test the semantic
segmentation capabilities of AI. This dataset
contains 25,000 annotated images of diverse urban

Figure 2.2.9

environments. The Cityscapes dataset enables avariety of different segmentation tasks. One of thegreater the mIoU, the better a system has performed.

most popular is the pixel-level task. Performanceon semantic segmentation is measured by mean

Performance on Cityscapes has increased by 23.4

intersection-over-union (mIoU), which represents thepercentage points since the competition launched indegree to which the image segments predicted by the

2014; however, it has plateaued in the last few years

model overlap with the image’s actual segments. The(Figure 2.2.10).

Cityscapes Challenge, Pixel-Level Semantic Labeling Task: Mean Intersection-Over-Union (mIoU)
Source: Cityscapes Challenge, 2022 | Chart: 2023 AI Index Report

86.46%, With Extra Training Data

Mean Intersection-Over-Union (mIoU)

85%

84.30%, Without Extra Training Data

80%

75%

70%

65%

2014

2015

2016

2017

2018

2019

2020

2021

2022

Figure 2.2.10

Chapter 2 Preview

18Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Medical Image
Segmentation

A Demonstration of Medical Imaging Segmentation
Source: Jha et al., 2019

In medical image segmentation, AI systems
segment objects such as lesions or organs inmedical images (Figure 2.2.11).

Kvasir-SEG
Kvasir-SEG is a dataset for medical image

Figure 2.2.11

segmentation that contains 1,000 highquality images of gastrointestinal polyps
that were manually identified by medical
professionals. Progress on Kvasir-SEG ismeasured in mean Dice, which represents
the degree to which the polyp segments

This year’s top-performing model on Kvasir-SEG, SEP, wasidentified by AI systems overlap with thecreated by a Chinese researcher and posted a mean Dice ofactual polyp segments.1

94.1% (Figure 2.2.12).

Kvasir-SEG: Mean Dice

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

94.11%

Mean Dice

90%

85%

2019

2020

2021

2022
Figure 2.2.12

1 Mean Dice and mIoU are in principle quite similar. This StackExchange post outlines the differences in more detail.

Chapter 2 Preview

19Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Object Detection

A Demonstration of Object Detection
Source: Rizzoli, 2023

The challenge of identifying and localizing objects
within an image or video is known as object
detection (Figure 2.2.13).

Common Objects in Context (COCO)
Microsoft’s Common Objects in Context (COCO)
object detection dataset has over 80 object
categories in 328,000 images. Several accuracy
metrics are used to measure progress on COCO.
This section considers mean average precision(mAP50).
Since 2015, state-of-the-art detectors have
improved by 26 percentage points. The top model

Figure 2.2.13

in 2022, EVA, was the result of a Chinese academic
research collaboration.

COCO: Mean Average Precision (mAP50)
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

81.90%

Mean Average Precision (mAP50)

80%

70%

60%

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.14

Chapter 2 Preview

20Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Which Face Is Real?

Image Generation

Source: Which Face Is Real?, 2022

Image generation is the task of generating images thatare indistinguishable from real ones. In the last decade,progress on image generation has tremendously
increased, so much so that now it would be difficult
for the average person to distinguish a real human facefrom one synthetically generated by AI (Figure 2.2.15).

CIFAR-10 and STL-10

Figure 2.2.15

CIFAR-10 and STL-10 are two popular benchmarks
for tracking progress on image generation. CIFAR-10
comprises 60,000 color images across 10 different

set of images is similar to the real images on which it
was trained.

object classes; STL-10 is inspired by CIFAR-10, withThis year saw state-of-the-art results on both CIFAR-10

some modifications, including fewer labeled training

and STL-10 benchmarks (Figure 2.2.15). The top

examples and more unlabeled examples. Progress onmodel on CIFAR-10, EDM-G++, came from Korean

image generation in both benchmarks is measured

researchers at KAIST. The top model on STL-10 wasby the Fréchet Inception Distance (FID) score, which

Diffusion-GAN, a collaboration between researchers at

reflects the degree to which a synthetically generated

the University of Texas at Austin and Microsoft.

CIFAR-10 and STL-10: Fréchet Inception Distance (FID) Score
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

35

Fréchet Inception Distance (FID) Score

30

25

20

15

10
6.91, STL-10
5
1.77, CIFAR-10
0
2017

2018

2019

2020

2021

2022
Figure 2.2.16

Chapter 2 Preview

21Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

A Closer Look at Progress in Image Generation
Figure 2.2.17 tracks the progress of facial

GAN Progress on Face Generation

image generation over time, with the final

Source: Goodfellow et al., 2014; Radford et al., 2016; Liu and Tuzel, 2016;
Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al., 2020;
Vahdat et al., 2021; Wang et al., 2022.

image being generated by Diffusion-GAN,
the model that posted the 2022 state-ofthe-art score on STL-10.

2014

2015

2016

2017

2018

2020
2021

2022
Figure 2.2.17

In the last year, text-to-image
generation broke into the public

Images Generated by DALL-E 2, Stable Diffusion and Midjourney
Source: AI Index, 2022

consciousness with the releaseof models such as OpenAI’s
DALL-E 2, Stability AI’s Stable
Diffusion, Midjourney’s
Midjourney, Meta’s Make-AScene, and Google’s Imagen.

a. DALL-E 2

With these systems, users cangenerate images based ona text prompt. Figure 2.2.18
juxtaposes the images generated
by DALL-E 2, Stable Diffusion,and Midjourney, three publicly
accessible AI text-to-image
systems, for the same prompt: “apanda playing a piano on a warm
evening in Paris.”

Chapter 2 Preview

b. Stable Diffusionc. Midjourney

Figure 2.2.18

22

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

A Closer Look at Progress in Image Generation (cont’d)
Of all the recently released text-to-image generators, Google’s Imagen performs best on the
COCO benchmark (Figure 2.2.19)2. This year, the Google researchers who created Imagen
also released a more difficult text-to-image benchmark, DrawBench, designed to challenge
increasingly capable text-to-image models.

Notable Text-to-Image Models on MS-COCO 256 × 256 FID-30K: Fréchet Inception Distance (FID) Score
Trained on COCO-FID

35.49

Not Trained on COCO-FID

32.64
30

21.42

20.79

20

17.89
12.24

2021

10.39

Make-A-Scene

DALL-E 2

7.55

7.27

Imagen

8.12

LAFITE

2020

GLIDE

2019

DALL-E

DF-GAN2017

DM-GAN + CL

DM-GAN

0

XMC-GAN

9.33

10

AttnGAN

COCO Fréchet Inception Distance (FID) Score

Source: Saharia et al., 2022 | Chart: 2023 AI Index Report

2022

Model

Figure 2.2.19

2 The COCO benchmark, first launched in 2014, includes 328,000 images with 2.5 million labeled instances. Although it is typically used for object detection tasks, researchers
have also deployed it for image generation.

Chapter 2 Preview

23Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Visual Reasoning
Visual reasoning tests how well AI systems can reason across both textual and visual data,
as in the examples of Figure 2.2.20.
A Collection ofVisual Reasoning
Tasks
Source: Agrawal et al., 2016

Figure 2.2.20

Visual Question Answering (VQA) Challenge
The Visual Question Answering Challenge tests AI
systems with open-ended textual questions about
images. Successfully answering the questions
requires that AI systems possess vision, language, andcommonsense reasoning capabilities. This section

reports progress on the VQA V2 dataset.
This year the top-performing model on VQA V2
was PaLI, a multimodal model produced by Google
researchers (Figure 2.2.21).

Visual Question Answering (VQA) V2 Test-Dev: Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report85%

84.30%

80.78%, Human Baseline

Accuracy (%)

80%

75%

70%

65%
2016

2017

2018

2019

2020

2021

2022
Figure 2.2.21

Chapter 2 Preview

24Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

The Rise of Capable Multimodal Reasoning Systems
Traditionally AI has been strong in narrow tasks,

models were introduced, for example BEiT-3 frombut it has been unable to easily generalize across

Microsoft and PaLI from Google, that posted state-

multiple domains. For instance, many imageof-the-art results across a variety of both vision andclassifiers are adept at classifying images but arelanguage benchmarks. For example, at the time ofincapable of understanding written text.

publication of the BEiT-3 paper, BEiT-3 posted state-

However, recent technical progress in AI has
begun to challenge this notion. In 2022, severalof-the-art results for four different vision skills andfive different vision-language skills (Figure 2.2.22).

BEiT-3Vs.
Vs. Previous State-of-the-Art
State-of-the-Art Models
ModelsBEiT-3
BEiT-3
Source: Wang et Previousal., 2022 | Table: 2023 AI Index Report
Source: Wang
Wang etet al.,
al., 2022
2022 || Table:
Table: 2023
2023 AI Index Report
Source:Category
Category
Category

Task
Task
Task

DatasetDatasetMetricMetricPrevious SOTA
Previous SOTA

VisionVisionVisionSemantic
Semantic
Semantic
Segmentation
Segmentation
Segmentation

ADE20K
ADE20K

mIoU
mIoU

61.40 FD-SwimV2
61.40 FD-SwimV2

62.80
62.80
62.80

2.28%
2.28%
2.28%

VisionVisionVisionObjectObjectObjectDetection
Detection
Detection

COCO
COCO

AP
AP

63.30 DINO
63.30 DINO

63.70
63.70
63.70

0.63%
0.63%
0.63%

VisionVisionVisionInstance
Instance
Instance
Segmentation
Segmentation
Segmentation

COCO
COCO

AP
AP

54.70 Mask DINO
54.70 Mask DINO

54.80
54.80
54.80

0.18%
0.18%
0.18%

VisionVisionVision

Image
Image
Image
Classi cation
Classi cation
ClassiImageNet
ImageNet

Top-1 Accuracy
Top-1 Accuracy

89.00 FD-CLIP
89.00 FD-CLIP

89.60
89.60
89.60

0.67%
0.67%
0.67%

Vision-Language Visual
Vision-Language Visual
Vision-Language
VisualReasoning
Reasoning
Reasoning
Vision-Language Visual QA
Vision-Language Visual
Visual QA
Vision-Language
Vision-Language Image
Vision-Language Image
Vision-Language
Image
Captioning
Captioning
Captioning

NLVR
NLVR

Accuracy
Accuracy

87.00 CoCA
87.00 CoCA

92.60
92.60
92.60

6.44%
6.44%
6.44%

VQAv2
VQAv2
COCO
COCO

VQA Accuracy
VQA Accuracy
CIDEr
CIDEr

82.30 CoCA
82.30 CoCA
145.30 OFA
145.30 OFA

84.00
84.00
84.00
147.60147.60147.60

2.07%
2.07%
2.07%
1.58%
1.58%
1.58%

Vision-Language Finetuned
Vision-Language Finetuned
Vision-Language
Finetuned
Retrieval
Retrieval
Retrieval

COCO
COCO
Flickr30K
Flickr30K

R@1
R@1

72.50 Florence
72.50 Florence

76.00
76.00
76.00

4.83%
4.83%
4.83%

Vision-Language Zero-Shot
Vision-Language Zero-Shot
Vision-Language
Zero-Shot
Retrieval
Retrieval
Retrieval

Flickr30K
Flickr30K
Flickr30K

R@1
R@1
R@1

86.50 CoCA
86.50
86.50 CoCA
CoCA

88.20
88.20
88.20

1.97%
1.97%
1.97%

Model of
Model ofPrevious SOTA
Previous SOTA

BEiT-3
BEiT-3

Scale of
Scale
Scale ofof
Improvement
Improvement
Improvement

Figure 2.2.22

Chapter 2 Preview

25Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

The Rise of Capable Multimodal Reasoning Systems (cont’d)Figure 2.2.23 shows some of the different vision-language tasks challenging multimodal systems like
PaLI and BEiT-3.
A Collection of Vision-Language Tasks
Source: Chen et al., 2022

Figure 2.2.23

Chapter 2 Preview

26Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Visual Commonsense Reasoning (VCR)

example of a question posed in VCR. Performance on

The Visual Commonsense Reasoning challenge, first

VCR is tracked in the Q->AR score, which combines

launched in 2019, is a relatively new benchmark in

the ability of machines to select the right answer

which AI systems must answer questions presented

for the question (Q->A) and the ability to select thefrom images, as in VQA, but also select the reasoning

correct rationale behind the answer (Q->R).

behind their answer choices. Figure 2.2.24 shows an
A Sample Question from the Visual Commonsense Reasoning (VCR) Challenge
Source: Zellers et al., 2018

Figure 2.2.24

VCR is one of the few visual benchmarks consideredin this report on which AI systems have yet to surpass
human performance, as shown in Figure 2.2.25.

Visual Commonsense Reasoning (VCR) Task: Q->AR Score
Source: VCR Leaderboard, 2022 | Chart: 2023 AI Index Report

85.00, Human Baseline

80
75.60

Q->AR Score

70

60

50

2018

2019

2020

2021

2022
Figure 2.2.25

Chapter 2 Preview

27Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

Video analysis concerns reasoning or task operation across videos, rather than single images.

2.3 Computer Vision—Video
Activity Recognition

Kinetics-400, Kinetics-600, Kinetics-700

Activity recognition is the categorization of activities

series of datasets for benchmarking video activity

that occur in videos. Certain activities, such asrecognition. Each dataset includes 650,000 large-

sitting, sleeping, or walking, are easier for AI systems

scale, high-quality video clips from YouTube that display

to categorize than others which involve multiple

a wide range of human activities, and each asks AI

steps—for example, preparing dinner.

systems to classify an action from a possible set of 400,

Kinetics-400, Kinetics-600, and Kinetics-700 are a

600, and 700 categories, respectively (Figure 2.3.1).
Example Classes From the Kinetics Dataset
Source: Kay et al., 2017

Figure 2.3.1

Chapter 2 Preview

28Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

As of 2022, there is a 7.8 percentage point gap in performance between the top system on Kinetics-600 andKinetics-700, which suggests the 700 series dataset is still a meaningful challenge for video computer visionresearchers (Figure 2.3.2).

Kinetics-400, Kinetics-600, Kinetics-700: Top-1 Accuracy
Source: Papers With Code, 2021; arXIv, 2022 | Chart: 2023 AI Index Report

91.80%, Kinetics-600
91.10%, Kinetics-400

90%Top-1 Accuracy (%)

84.00%, Kinetics-700
80%

70%

60%

2016

2017

2018

2019

2020

2021

2022
Figure 2.3.2

Chapter 2 Preview

29Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

Narrative Highlight:

A Closer Look at the Progress of Video Generation
Multiple high quality text-to-video models,In September 2022, CogVideo’s top score was

AI systems that can generate video clips fromsignificantly surpassed by Meta’s Make-A-Video

prompted text, were released in 2022 . In May,model (Figure 2.3.3). Make-A-Video performed

researchers from Tsinghua University and the

63.6% better on UCF-101 than CogVideo. And, in

Beijing Academy of Artificial Intelligence released

October 2022, Google released a text-to-video

CogVideo, a model that posted the then-highest

system called Phenaki; however, this model was

inception score on the UCF-101 benchmark for

not benchmarked on UCF-101.

3

text-to-video generation (Figure 2.3.3).

Notable Text-to-Video Models on UCF-101: Inception Score (IS)
Source: Hong et al., 2022; Singer et al., 2022 | Chart: 2023 AI Index Report

79.28

80

82.55

70

Inception Score (IS)

60
50.46

50
40
32.7

32.36
30

27.38

28.87
24.69

20
10
0

DVD-GAN2019

TGANv2
2020

VideoGPT

MoCoGAN-HD
2021

DIGAN

CogVideo

TATS-baseMake-A-Video

2022

Model

Figure 2.2.3

3 Although these models are impressive, it is worth noting that they are thus far only capable of generating videos of a few seconds’ duration.

Chapter 2 Preview

30Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Natural language processing (NLP) is the ability of computer systems to understand text. The last few years have seen the release ofincreasingly capable “large language models,” AI systems like PaLM, GPT-3, and GLM-130B, that are trained on massive amounts of data
and adaptable to a wide range of downstream tasks.
In this section, progress in NLP is tracked across the following skill categories: (1) English language understanding, (2) text summarization,(3) natural language inference, (4) sentiment analysis, (5) multitask language understanding, and (6) machine translation.

2.4 Language
English Language
Understanding

SuperGLUE

English language understanding challenges AI

of AI models on eight different linguistic tasks.

systems to understand the English language in

A selection of these tasks is highlighted in Figure

various ways: reading comprehension, yes/no2.4.1. Their performance is then aggregated into areading comprehension, commonsense reading

single metric.

SuperGLUE is a comprehensive English language
understanding benchmark that tracks the progresscomprehension, and logical reasoning.
A Set of SuperGLUE Tasks4
Source: Wang et al., 2019

Figure 2.4.1

4 For the sake of brevity, this figure only displays four of the eight tasks.

Chapter 2 Preview

31Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

This year’s top model on SuperGLUE, Vega, registered a new state-of-the-art score of 91.3, which is 1.5
percentage points higher than the human baseline. Performance on SuperGLUE is continuing to saturate.

SuperGLUE: Score

Source: SuperGLUE Leaderboard, 2022 | Chart: 2023 AI Index Report

91.30
91

90

89.80, Human Baseline

Score

89

88

87

86

85

2019

2020

2021

2022
Figure 2.4.2

Reading Comprehension
Dataset Requiring Logical
Reasoning (ReClor)
In response to the saturation of
traditional reading comprehension
benchmarks, researchers from the
National University of Singapore
launched ReClor in 2020. ReClor,
or Reading Comprehension Dataset
Requiring Logical Reasoning, is adataset of logical reasoning questions
taken from the LSAT, the entrance
exam for law schools in the UnitedStates and Canada. A sample
question is shown in Figure 2.4.3

A Sample Question from the Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)
Source: Yu et al., 2020

Context: When a certain gland becomes cancerous in humans, it produces high levelsof a particular protein. A blood test can determine the level of this protein well before
a cancer of the gland could be detected by other means. Some doctors recommendthat aggressive anticancer treatment should be begun as early as possible for anyone
who is tested and is found to have high levels of the protein.
Question: Which one of the following, if true, most seriously weakens the doctors’
recommendation?
A. The blood test for the protein has been in use for some time to monitor thecondition of patients who have been diagnosed as having cancer of the gland.
B. Before the blood test became available, about one-third of all cases of cancer ofthe gland were detected in early stages.
C. So far, no patients whose protein levels were found to be normal havesubsequently developed cancer of the gland.
D. Enlargement of the gland, a common condition infrequently associated withcancer, results in high levels of the protein.

Figure 2.4.3

Chapter 2 Preview

32Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Figure 2.4.4 examines progress on ReClor. The top 2022 result of 80.6% represented an 18 percentage point
improvement from 2020, the year the benchmark was released.

Reading Comprehension Dataset Requiring Logical Reasoning (ReClor): Accuracy
Source: ReClor Leaderboard, 2022; Papers With Code, 2022 | Chart: 2023 AI Index Report

80.60%

80%

Accuracy (%)

75%

70%

65%

2020

2021

2022
Figure 2.4.4

Chapter 2 Preview

33Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Narrative Highlight:

Just How Much Better Have Language Models Become?
The AI Index tested how three large language models from three different years, GPT-2 (2019), GPT-3(2020), and ChatGPT (2022), handle the same prompt: “Explain to me the major accomplishments of
Theodore Roosevelt’s presidency.” More recent models are able to answer this question more effectively,
both in terms of factual accuracy and quality of writing.
GPT-2 (February 2019)5

GPT-3 (June 2020)7

ChatGPT (November 2022)(The most notable is the establishment of anational telephone and mail exchange company.)

Theodore Roosevelt’s presidency
was marked by a number ofaccomplishments. He is credited withleading the United States out of theGreat Depression and into World War II.
He also spearheaded the creation of the
National Park Service and the National
Wildlife Federation. Roosevelt also
played a major role in the developmentof the Panama Canal.

Theodore Roosevelt, President from 1901-1909,
accomplished the following major accomplishments
during his presidency:

Commentary: This answer is an
improvement over that of GPT-2 inthat it actually somewhat responds tothe question, but still shows significant
flaws. GPT-3 is correct in asserting that
Theodore Roosevelt was instrumentally
involved in the development of the
Panama Canal. However, he did notlead the United States out of the Great
Depression and into World War II—that
was done by a different Americanpresident, Franklin Delano Roosevelt.
It appears that GPT-3 confusedsome of Franklin Delano Roosevelt’s
accomplishments with those of
Theodore Roosevelt. Finally, although
Theodore Roosevelt did play a significant
role in establishing many national parks,
he did not in fact create the National
Park Service or the National Wildlife
Federation. The National Park Service
was created in 1916 by Woodrow Wilson,
and the National Wildlife Federationin 1936, much later than Theodore
Roosevelt’s presidency. 8

3. Labor Reforms: Improved working conditions
for Americans through advocacy for better hours,
wages, and workplace safety.

But the only things that will change with apresident who knows how to operate a businessis his understanding of the American systemof government. The public knows all about our
problems, yet at least half of the people whoneed it won’t know anything about ours. We canno longer have this. Our system of governmentis the only one in which you really need a public
relations firm to work hard.
The system of governmental control, whileworking, is not always effective in ensuring thatthe very things that our economy needs, do
not always be at the top of everybody’s list. So
I wanted to know how successful the systemyou employ for managing the public’s needs isin helping the American economy grow, howefficient the service our economy provides isbeing made, and how it can better serve theneeds of a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer
to work on the case. This was before I everhired my own law firm to handle government
investigations. For my first major job, at least,
I had to convince two top aides to get paid
$90,000 for the role, and before I even started on
the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with mylead prosecutor...6
Commentary: Most of this answer is gibberish.
GPT-2 is able to respond with sentences that arelargely grammatical, but fails to actually answer
the question in any meaningful way.

1. Conservation: Established numerous nationalparks, monuments, and forests, including the Grand
Canyon National Park.
2. Trust-busting: Took on monopolies, such as
Standard Oil, through antitrust laws and broke uplarge corporations.

4. Progressive Reforms: Promoted progressive
policies, including regulation of big business andprotection of natural resources.
5. National Infrastructure: Improved national
infrastructure through construction of the Panama
Canal and funding for dam projects.
6. Diplomacy: Negotiated the end of the RussoJapanese War, resulting in his being awarded theNobel Peace Prize.
7. Foreign Policy: Established the Roosevelt
Corollary, asserting U.S. intervention in Latin America
to maintain stability and protect American interests.
Commentary: ChatGPT’s response is the strongestof the three. All but one of the facts about Theodore
Roosevelt’s presidency are correct. Roosevelt
proclaimed the Grand Canyon Game Preserve in1906 and designated it a national monument in
1908. However, the Grand Canyon National Park
was itself created by Woodrow Wilson.9 Otherwise
the response is factually accurate. The response isalso more thorough and concise than the answers
provided by GPT-2 and GPT-3.

5 GPT-2 used the 124M parameter model downloaded from OpenAI’s GitHub page.
6 The complete answer outputted by GPT-2 is trimmed here for brevity. The full answer is included in the Appendix.
7 The specific GPT-3 model that was used was text-curie-001, which has training data up to October 2019.
8 The information in this section has been cross-verified with the Encyclopedia Britannica entries on Theodore Roosevelt, Franklin Delano Roosevelt, Woodrow Wilson, and the
National Park Service, as well as the history page of the National Wildlife Federation.
9 Information on the history of the Grand Canyon National Park was cross-verified with the Wikipedia entry on the Grand Canyon National Park.

Chapter 2 Preview

34Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Narrative Highlight:

Planning and Reasoning in Large Language Models
As illustrated above, AI systems have become

The authors then tested notable language models

increasingly strong on a wide range of reasoning

on these tasks in a Blocksworld problem domain,tasks. This improvement has led many to claim thata problem environment where agents are given

emerging AI systems, especially large language

blocks of different colors and tasked with arranging

models, possess reasoning abilities that arethese blocks in particular orders. The authors

somewhat similar to those possessed by humans.

demonstrated that these large language models

Other authors, however, have argued otherwise.

performed fairly ineffectively (Figure 2.4.5). While

10

11

GPT-3, Instruct-GPT3, and BLOOM demonstrated

In 2022, researchers (Valmeekam et al., 2022)
introduced a more challenging planning and reasoning
test for large language models that consists of seven

the ability, in some contexts, to reformulate goals
in robust ways, they struggled with other tasks likeplan generation, optimal planning, and plan reuse.

assignments: (1) plan generation, (2) cost-optimal

Compared to humans, the large language models

planning, (3) reasoning about plan execution, (4)performed much worse, suggesting that while theyrobustness to goal reformulation, (5) ability to reuse

are capable, they lack human reasoning capabilities.

plans, (6) replanning, and (7) plan generalization.12

Select Large Language Models on the Blocksworld Domain: Instances Correct
Source: Valmeekam et al., 2022 | Chart: 2023 AI Index Report

0.6%

Plan Generation

GPT-3

5.0%

Instruct-GPT3

0.5%

Optimal Planning

BLOOM

0.2%
3.2%
0%
5.6%
4.8%
3.0%

Replanning

6.6%

Plan Generalization

9.8%
11.0%
0%

Plan Reuse

14.4%
0%
77.4%
76.8%

Robustness to Goal Reformulation(Shu ing Goal Predicates)

21.0%
69.2%

Robustness to Goal Reformulation(Full → Partial)
Robustness to Goal Reformulation(Partial → Full)
0%

76.0%
9.0%
22.0%
60.2%
5.0%

10%

20%

30%

40%
50%
Instances Correct (%)

60%

10 Some of the papers that claim language models can reason include: Kojima et al., 2022; Chowdhery et al., 2022; Li et al., 2021; Wei et al., 2022.
11 Valmeekam et al., 2022 advances this claim.
12 A complete description of these tasks can be found in the paper.

Chapter 2 Preview

70%

80%

Figure 2.4.5

35

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Text Summarization
Text summarization tests how well AI systems canarXiv and PubMed

synthesize a piece of text while capturing its core

ArXiv and PubMed are two widely used datasets forcontent. Text summarization performance is judged

benchmarking text summarization. The model thaton ROUGE (Recall-Oriented Understudy for Gisting

posted the state-of-the-art score in 2022 on both

Evaluation), which measures the degree to which

arXiv and PubMed, AdaPool, was developed by aan AI-produced text summary aligns with a humanteam from Salesforce Research (Figure 2.4.6).

reference summary.

ArXiv and PubMed: ROUGE-1

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

51.05, PubMed
50.95, arXiv
50

ROUGE-1

45

40

35

2017

2018

2019

2020

2021

2022
Figure 2.4.6

Chapter 2 Preview

36Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Natural Language Inference

uncertain premises. Imagine, for example, that Peter

Also known as textual entailment, natural language

window shattered and his laptop, which he left in theinference is the ability of AI systems to determine
whether a hypothesis is true, false, or undeterminedbased on presented premises.

Abductive Natural Language Inference (aNLI)
Abductive natural language inference is a formof natural language inference in which plausible
conclusions must be drawn from a set of limited andreturns to his car after dinner at a restaurant to find theback seat, missing. He might immediately conclude
that a thief broke into his car and stole the laptop.
In 2019, the Allen Institute for AI launched aNLI, a
comprehensive benchmark for abductive natural
language inference that includes 170,000 premiseand hypothesis pairs (Figure 2.4.7).

Sample Question From
the Abductive Natural
Language Inference
Benchmark (aNLI)
Source: Allen Institute for AI, 2021

Figure 2.4.7

Abductive natural language inference is a challenging task. The human baseline remained
unsurpassed until 2022, when an AI system registered a score of 93.7% (Figure 2.4.8).

Abductive Natural Language Inference (aNLI): Accuracy
Source: Allen Institute for AI, 2022 | Chart: 2023 AI Index Report

94%93.65%
92.90%, Human Baseline

Accuracy (%)

92%

90%

88%

86%

84%
2019

2020

2021

2022
Figure 2.4.8

Chapter 2 Preview

37Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Sentiment Analysis

A Sample Sentence from SST
Source: Socher et al., 2013

Sentiment analysis applies NLP techniques to
identify the sentiment of a particular text. It is usedby many businesses to better understand customer
reviews.

SST-5 Fine-Grained Classification
The Stanford Sentiment Treebank (SST) is a datasetof 11,855 single sentences taken from movie reviews
that are then transformed into 215,154 unique phrases
whose sentiments have been annotated by humanjudges (Figure 2.4.9).

Figure 2.4.9

A new state-of-the-art score of 59.8% was posted on SST-5 fine-grained classification by the
Heinsen Routing + RoBERTa Large model (Figure 2.4.10).

SST-5 Fine-Grained: Accuracy

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report60%

59.80%

Accuracy (%)

55%

50%

45%
2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.4.10

Chapter 2 Preview

38Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Multitask Language
Understanding

Sample Questions From MMLU
Source: Hendrycks et al., 2021

a) Sample Math Questions

A common criticism of language benchmarks
such as GLUE and SuperGLUE is that they do notaccurately test how capable language models are atapplying the knowledge they learn across different
domains.13 Multitask language understanding tests
the ability of language models to reason acrossspecialized subject domains.

Massive Multitask Language Understanding(MMLU)b) A Sample Microeconomics Question

Massive Multitask Language Understanding (MMLU)
evaluates models in zero-shot or few-shot settings
across 57 diverse subjects in the humanities, STEM,
Figure 2.4.11

and the social sciences (Figure 2.4.11).

Gopher, Chinchilla, and variants of PaLM have each posted state-of-the-art results on MMLU. The current top
result on MMLU comes from Flan-PaLM, a Google model that reports an average score of 75.2% (Figure 2.4.12).

MMLU: Average Weighted Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

75.20%
70%

Accuracy (%)

60%

50%

40%

30%

2019

2020

2021

2022
Figure 2.4.12

13 This criticism is more formally articulated in Hendrycks et al., 2021.

Chapter 2 Preview

39Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Machine Translation (MT)

Number of Commercially Available MT Systems

Machine translation studies how well AI software

The popularity of AI-based machine translation is

can translate languages. In the last five years,

manifested in the number of commercial machinemachine translation has been dominated by neuraltranslation services on the market. Since 2017, the total

networks which power current tools like DeepL andnumber of independent machine translation services

Google Translate.

has increased six times (Figure 2.4.13).

Number of Independent Machine Translation Services
Source: Intento, 2022 | Chart: 2023 AI Index Report

Commercial

80

Open Source Pre-trained

Number of Independent Machine Translation Services

Preview
70

60
54
50

46

40
34
45

30

26
21

20

10

0

23
28

16
9

10

8

9

2017-May

2017-Jul

12

13

9

10

2017-Nov

2018-Mar

18

38

21

23

15

2018-Jul

2018-Dec

2019-Jun

2019-Nov

65

5

2020-Jul

2021-Sep

2022-Jul
Figure 2.4.13

Chapter 2 Preview

40Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.5 Speech

AI systems that work with human speech are usually tasked with converting spoken words into text and recognizing the individuals speaking.

2.5 Speech
Speech Recognition

task of matching certain speech with a particular

Speech recognition is the ability of AI systems tobeen expanded; however, the data in this subsection

identify spoken words and convert them into text.
Speech recognition has progressed so much sothat nowadays many computer programs or texting
apps are equipped with dictation devices that canseamlessly transcribe speech into writing.

individual. Over the years, the VoxCeleb dataset has
tracks progress on the original dataset.
This year’s top result on the original VoxCeleb dataset
was posted by American researchers, whose model
achieved an equal error rate of 0.1%, which represents
a 0.28 percentage point decrease from the state-of–

VoxCeleb
VoxCeleb is a large-scale audiovisual dataset ofhuman speech for speaker recognition, which is thethe-art result achieved by Chinese researchers in theprevious year (Figure 2.5.1).

VoxCeleb: Equal Error Rate (EER)
Source: VoxCeleb, 2022 | Chart: 2023 AI Index Report8%

Equal Error Rate (EER)

6%

4%

2%

0.14%

0%
2017

2018

2019

2020

2021

2022
Figure 2.5.1

Chapter 2 Preview

41Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.5 Speech

Narrative Highlight:

Whisper

One of the major themes in the last few years of AI progress has been the emergence of large language
models that are trained on massive amounts of data and capable of executing a diverse range of tasks.
In 2022, this idea of training on large data to achieve cross-domain performance arrived in the world ofspeech recognition with OpenAI’s launch of Whisper.
Whisper is a large-scale speech recognition model that was trained in a weakly supervised way
on 700,000 hours of audio data. Whisper was capable of strong, although not state-of-the-art,
performance on many speech recognition tasks in zero-shot settings.14 Whisper outperformed wav2vec
2.0 Large, another speech recognition model, across a wide range of popular English speech recognition
benchmarks (Figure 2.5.2). Similarly, Whisper proved to be a better speech translator than many otherleading AI translator models (Figure 2.5.3). Whisper also outperformed other commercial automatedspeech recognition systems and scored similarly to top human transcription services (Figure 2.5.4).15
Despite this impressive performance, there were still some speech tasks, like language identification, on
which Whisper trailed state-of-the-art models (Figure 2.5.5).
wav2vec 2.0 Large (No LM) Vs. Whisper Large V2
Across Datasets

Notable Models on X→EN Subset of CoVoST 2
Source: Radford et al., 2022 | Chart: 2023 AI Index Report

Source: Radford et al., 2022 | Chart: 2023 AI Index Report
6.2%
5.2%

LibriSpeech Other

Zero-Shot Whisper

WSJ

36.4%
7.7%
3.9%
34.8%

CallHome

17.6%

MAESTRO

25.2%

mSLAM-CTC (2B)

24.8%

28.3%

Switchboard

13.8%
37.0%

AMI IHM

16.9%
35.6%

CORAAL

16.2%
17.9%

VoxPopuli En

7.3%
65.8%

CHiME-6

25.5%
10.5%

TED-LIUM

4.0%

FLEURS En

4.4%

XLS-R (2B)

22.1%

14.6%
29.9%

Common Voice

9.0%
24.5%

Artie
LibriSpeech Clean

29.1%

67.6%

AMI SDM1

6.2%

0%

wav2vec 2.0 Large (No LM)

XMEF-X

14.7%

Whisper Large V2

2.7%
2.7%

10%

20%

30%
40%
50%
Word Error Rate (%)

60%

70%

Figure 2.5.2

0%

10%
20%
30%
Bilingual Evaluation Understudy (BLEU) Score

Figure 2.5.3

14 Zero-shot learning refers to the ability of an AI system to learn a particular task without being trained on that task.
15 Kincaid46 is a dataset of 46 audio files and transcripts that were published in the blog post, “Which automatic transcription service is the most accurate?—2018.”

Chapter 2 Preview

42Artificial IntelligenceIndex Report 2023

Narrative Highlight:

Chapter 2: Technical Performance
2.5 Speech

Whisper (cont’d)

Notable Speech Transcription Services on Kincaid46
Source: Radford et al., 2022 | Chart: 2023 AI Index Report
ASR

Computer-Assisted

9.66%

Company B

9.74%

Company C

10.90%

Company D

12.20%

Company E

7.61%

Company F

8.14%

Company G

8.65%

Company H

8.96%

Company I

Language Identi cation Accuracy (%)

8.81%

Company A

0%Source: Radford et al., 2022 | Chart: 2023 AI Index Report

Human Transcription

Whisper

Notable Models on FLEURS: Language Identi cation
Accuracy

77.7%

80%
71.4%

64.5%
60%

40%

20%

10.50%
2%

4%
6%
8%
10%
Median Word Error Rate (%)

12%

14%

0%

w2v-bert-51 (0.6B)

Figure 2.5.4

mSLAM-CTC (2B)

Zero-shot Whisper

Figure 2.5.5

Whisper represents a breakthrough in state-of-the-art speech recognition systems. Traditionally, such
systems were either pre-trained using supervised learning methods or pre-trained without supervision
but required fine-tuning. Acquisition of data for supervised pre-training is time-consuming and costly.
However, pre-training without supervision still requires further algorithmic specification to realize a desired
objective like speech recognition. Algorithmic specification itself often requires a skilled practitioner.
Whisper resolves these issues by demonstrating that a speech recognition system can perform well acrossa diverse range of tasks with massive amounts of unlabeled speech data.

Chapter 2 Preview

43Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learning from their prioractions. Systems are rewarded if they achieve a desired goal and punished if they fail.

2.6 Reinforcement Learning
Reinforcement Learning
Environments

The Different Environments in Procgen
Source: OpenAI, 2019

Reinforcement learning agents require environments,
not datasets, to train: They must be trained inenvironments where they can experiment withvarious actions that will allow them to identify
optimal game strategies.

Procgen
Procgen is a reinforcement learning environment
introduced by OpenAI in 2019. It includes
16 procedurally generated video-game-likeenvironments specifically designed to test the

Figure 2.6.1

ability of reinforcement learning agents to learn
generalizable skills (Figure 2.6.1). Performance on
Procgen is measured in terms of mean-normalized
score. Researchers typically train their systems on
200 million training runs and report an average scoreacross the 16 Procgen games. The higher the systemscores, the better the system.

Chapter 2 Preview

44Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

A team of industry and academic researchers from Korea posted the top score of 0.6 on Procgen in 2022 (Figure 2.6.2).

Procgen: Mean of Min-Max Normalized Score
Source: arXiv, 2022 | Chart: 2023 AI Index Report

Mean of Min-Max Normalized Score

0.57

0.50

0.40

2019

2020

2021

2022
Figure 2.6.2

Chapter 2 Preview

45Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

Narrative Highlight:

Benchmark Saturation
An emerging theme in this year’s AI Index is the observed performance saturation across many popular
technical performance benchmarks. Last year’s AI Index Report observed a similar trend; however,benchmark saturation has been particularly pronounced this year. Figure 2.6.3 shows the relative
improvement since the benchmark first launched (overall improvement) and relative improvement within
the last year (YoY improvement) on AI technical benchmarks considered in this year’s AI Index. The
improvements are reported as percent changes.
For all but 7 of the benchmarks, the improvement registered is less than 5%. The median improvement
within the last year is 4%, while the median improvement since launch is 42.4%.16 Moreover, this year the
AI Index elected not to feature traditionally popular benchmarks like SQuAD1.1 and SQuAD2.0, as no
new state-of-the-art results were posted. Moreover, the speed at which benchmark saturation is being
reached is increasing. Researchers have responded to this increasing saturation by launching newer andmore comprehensive benchmarking suites such as BIG-bench and HELM.
Improvement Over Time on Select AI Index Technical Performance Benchmarks

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Overall Improvement
YoY Improvement

120%

Improvement (%)

100%
80%
60%
40%

MMLU

SST-5

ProcgenLanguage

ANLI

PubMedarXiv

ReClorSuperGLUE

Kinetics-700

Kinetics-600

Kinetics-400

COCO

VQA

CIFAR-10

VCR

Vision Video

VoxCeleb

Vision Image

STL-10Kvasir-SEG

Cityscapes

MPII

Celeb-DFFVRT

0%ImageNet Top-1

20%

SR

RL

BenchmarkFigure 2.6.3
16 The improvements reviewed in this section are reported as relative change. Figure 2.6.3 should therefore not be used to conduct comparisons of improvements acrossbenchmarks, as each benchmark has different parameters.

Chapter 2 Preview

46Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Deep learning AI algorithms are trained on GPUs or TPUs, which accelerate the training speed of AI systems. As AI systems process
ever-larger datasets, it is crucial to monitor advancements in hardware capabilities.

2.7 Hardware
MLPerf Training

every AI skill category had significantly decreased.

MLPerf is an AI training competition run by theThis year, this trend has continued, albeit at a slightly

ML Commons organization. In this challenge,

slower pace. Record-low training times were posted

participants train ML systems to execute various

in the object detection, speech recognition, image

tasks using a common architecture. Entrants are thensegmentation, recommendation, image classification,

ranked on their absolute wall clock time, which isand language processing categories (Figure 2.7.1).

how long it takes for the system to train.

In categories like image classification and object

Last year, the AI Index observed that since thecompetition launched, training times for virtuallydetection, the top AI systems can now train roughly
32 times quicker than in 2018, when the competition
first launched.

MLPerf Training Time of Top Systems by Task: Minutes
Source: MLPerf, 2022 | Chart: 2023 AI Index Report60

Training Time (Minutes; Log Scale)

40
20
10
5
2.25, Object Detection (Heavyweight)
2.15, Speech Recognition

2

1.22, Image Segmentation

1

0.52, Recommendation

0.5

0.34, Object Detection (Lightweight)
0.2

0.19, Image Classi cation
0.18, Language Processing
2018

2019

2020

2021

2022
Figure 2.7.1

Chapter 2 Preview

47Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Data on the number of accelerators used by the

between the mean number of accelerators used byhardware systems submitted to MLPerf also

all entrants and the average accelerators used by thesuggests that stronger hardware has been poweringsystems that post the top results.17 This gap suggests

decreasing training times (Figure 2.7.2). Since thethat having better hardware is essential to training thestart of the MLPerf competition, the gap has grown

fastest systems.

MLPerf Hardware: Accelerators
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

4,500
4,216, Maximum Number of Accelerators Used
4,000

Number of Accelerators

3,500
3,000
2,500
2,000

1,859, Average Accelerators Used by Top System

1,500
1,000
500
211, Mean Number of Accelerators
0
20
18-

De

c-1

20
192

20

Ju

n-1

20

0

20

-Ju

l-2

21-

9

20

Ju

n-3

21-

0

20

De

c-0
1

22

20

-Ju

22

n-2

9

-N

ov
-09
Figure 2.7.2

17 An accelerator, like a GPU or TPU, is a chip that is chiefly used for the machine learning component of a training run.

Chapter 2 Preview

48Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

MLPerf Inference

Figures 2.7.3 to 2.7.6 plot the throughput of the state-of-

In deploying AI, inference is the step where trained

the-art submissions on MLPerf Inference across four skill

AI systems generate predictions, e.g. classifying

categories: image classification, language processing,objects.

recommendation, and speech recognition. The number of

In 2020, ML Commons introduced MLPerf Inference,a performance benchmarking suite that measures
how fast a trained AI system can process inputs andproduce inferences. The MLPerf Inference suite
tracks the throughput of AI systems, measured insamples per second or queries per second.18
MLPerf Best-Performing Hardware for ImageClassi�cation: O�ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report700k

inferences generated by the top-performing AI systems
has significantly increased since the first iteration of thecompetition in 2020. For example, the number of offline
samples generated by the top image classifiers andlanguage processors have more than doubled since 2020,
while those for recommendation systems have increased
by roughly 23%.
MLPerf Best-Performing Hardware for Language
Processing: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

679,915, O ine (Samples/s)

650k

75,153, O ine (Samples/s)
70,992, Server (Queries/s)

70k

630,221, Server (Queries/s)

600k
60k
Throughput

Throughput

550k
500k
450k
400k

50k

40k

350k
30k

300k
250k
2020

2021

2022

Figure 2.7.3

MLPerf Best-Performing Hardware for
Recommendation: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

2020

2022

Figure 2.7.4

MLPerf Best-Performing Hardware for Speech
Recognition: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

160k

2.7M

155,811, O ine (Samples/s)

2,683,620, Server (Queries/s)
2,645,980, O ine (Samples/s)

150k

2.6M

140k

136,498, Server (Queries/s)

130k
Throughput

2.5M
Throughput

2021

2.4M

120k
110k
100k

2.3M

90k
2.2M
80k
2.1M

70k
2020

2021

2022

Figure 2.7.5

2020

2021

2022

Figure 2.7.6

18 The following blog post from Dell Technologies offers a good distinction between offline and server samples: “Offline—one query with all samples is sent to the system under test (SUT).
The SUT can send the results back once or multiple times in any order. The performance metric is samples per second. Server—the queries are sent to the SUT following a Poisson distribution(to model real-world random events). One query has one sample. The performance metric is queries per second (QPS) within the latency bound.”

Chapter 2 Preview

49Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Trends in GPUs: Performance and Pricethe performance of a computational device. The higher

This year, the AI Index built on work previously

the FLOP/s, the better the hardware.

done by the research collective Epoch and analyzed
trends over time in GPU performance and price.19

Figure 2.7.8 showcases the median single performanceof new GPUs by release date, which continues to rise

Figure 2.7.7 showcases the FP32 (single precision)year over year. Since 2021, the median FLOP/s speed

performance FLOP/s of different GPUs released

has nearly tripled, and since 2003 it has increased

from 2003 to 2022. FLOP/s stands for “Floating

roughly 7,000 times.

Point Operations per second” and is a measure of

FP32 (Single Precision) Performance (FLOP/s) by
Hardware Release Date, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

Median FP32 (Single Precision) Performance (FLOP/s),
2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

2.0e+141.0e+145.0e+13

1.0e+132.0e+13
1.0e+135.0e+125.0e+12Median FLOP/s (Log Scale)

2.0e+121.0e+125.0e+112.0e+111.0e+115.0e+102.0e+101.0e+105.0e+92.0e+121.0e+125.0e+112.0e+111.0e+115.0e+102.0e+101.0e+102.0e+91.0e+95.0e+85.0e+9

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

2.0e+9
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

FLOP/s (Log Scale)

2.23e+13

2.0e+13

Figure 2.7.7

Figure 2.7.8

19 The Appendix fully delineates both the methodology of this approach and the unique ways in which AI Index research built upon the existing Epoch research.

Chapter 2 Preview

50Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Finally, figures 2.7.9 and 2.7.10 consider GPU trends

2022 is 1.4 times greater than it was in 2021 andin terms of FLOP/s per U.S. Dollar. This statistic

5600 times greater than in 2003, showing a doubling

considers whether the underlying performance ofin performance every 1.5 years. As noted in similarGPUs is increasing relative to their changing costs.

analyses, improvements in the price–performance ofAs evidenced most clearly in Figure 2.7.10, theAI hardware has facilitated increasingly larger training

price–performance of GPUs is rapidly increasing.

runs and encouraged the scaling of large AI models.

20

The median FLOP/s per U.S. Dollar of GPUs in

FP32 (Single Precision) Performance (FLOP/s) per
U.S. Dollar by Hardware Release Date, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

Median FP32 (Single Precision) Performance (FLOP/s)
per U.S. Dollar, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

3.59e+10

35B

50B

Median FLOP/s per U.S. Dollar

30B
FLOP/s per U.S. Dollar

40B

30B

20B

10B

25B
20B
15B
10B
5B

0

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

0

Figure 2.7.9

Figure 2.7.10

20 The data in figures 2.7.9 and 2.7.10 has been adjusted for inflation. The exact details of the adjustment are outlined in greater detail in the Appendix.

Chapter 2 Preview

51Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

There have been mounting concerns about the environmental impact of computational resources and the energy required for AI
training and inference. Although there is no standard benchmark for tracking the carbon intensity of AI systems, this subsection
synthesizes the findings of different researchers who are exploring the link between AI and the environment. Conducting researchon the environmental effects of AI was challenging as there are wildly varying estimates, the validity of which have not yet been
definitively established. To that end, the AI Index focuses on research from a recent paper by Luccioni et al., 2022. As AI models
continue growing in size and become more universally deployed, it will be increasingly important for the AI research community toconsciously monitor the effect AI systems have on the environment.

2.8 Environment
Environmental Impact of Select Large
Language Models

challenging to directly compare the carbon footprint

Many factors determine the amount of carbon

reporting carbon emissions are not standardized.

of these models, as the accounting methodologies foremissions emitted by AI systems, including thenumber of parameters in a model, the power usage
effectiveness of a data center, and the grid carbon
intensity. Power Usage Effectiveness (PUE) is ametric used to evaluate the energy efficiency ofOf the four language models being compared, GPT3 released the most carbon, 1.4 times more thanGopher, 7.2 times more than OPT, and 20.1 times morethan BLOOM.

data centers. It is the ratio of the total amount of

Figure 2.8.2 relativizes the carbon-emission estimatesenergy used by a computer data center facility,

to real-life examples. For instance, BLOOM’s training

including air conditioning, to the energy delivered

run emitted 1.4 times more carbon than the average

to computing equipment. The higher the PUE, theAmerican uses in one year and 25 times that of flying

less efficient the data center. Figure 2.8.1 shows how

one passenger round trip from New York to San

these factors compare across four large language

Francisco. BLOOM’s training consumed enough energy

models: GPT-3, Gopher, OPT, and BLOOM. It is

to power the average American home for 41 years.21

Environmental Impact of Select Machine Learning Models, 2022
Source: Luccioni et al., 2022 | Table: 2023 AI Index Report

Model

Number of
Parameters

Datacenter PUE

Grid Carbon
Intensity

Power
Consumption

C02 Equivalent
Emissions

C02 Equivalent
Emissions x PUE

Gopher

280B

1.08

330 gC02eq/kWh

1,066 MWh

352 tonnes

BLOOM

176B

1.20

57 gC02eq/kWh

433 MWh

25 tonnes

380 tonnes
30 tonnes

GPT-3

175B

1.10

429 gC02eq/kWh

1,287 MWh

502 tonnes

552 tonnes

OPT

175B

1.09

231 gC02eq/kWh

324 MWh

70 tonnes

76.3 tonnes

Figure 2.8.1

21 The U.S. Energy Information Administration estimates that in 2021, the average annual electricity consumption of a U.S. residential utility customer was 10,632 kilowatt hours (kWh).

Chapter 2 Preview

52Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

CO2 Equivalent Emissions (Tonnes) by Selected Machine Learning Models and Real Life Examples, 2022
Source: Luccioni et al., 2022; Strubell et al., 2019 | Chart: 2023 AI Index Report

GPT-3 (175B)

502

Gopher (280B)

352

OPT (175B)

70

Car, Avg. Incl. Fuel,
1 Lifetime

63

BLOOM (176B)

25

American Life,
Avg., 1 Year

18.08

Human Life,
Avg., 1 Year

5.51

Air Travel,
0.99
1 Passenger, NY–SF
0

Chapter 2 Preview

50

100

150

200
250
300
CO2 Equivalent Emissions (Tonnes)

350

400

450

500
Figure 2.8.2

53

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

Narrative Highlight:

Using AI to Optimize Energy Usage
Training AI systems can be incredibly energy intensive. At the same time, recent research suggeststhat AI systems can be used to optimize energy consumption. In 2022, DeepMind released theresults of a 2021 experiment in which it trained a reinforcement learning agent called BCOOLER(BVE-based COnstrained Optimization Learner with Ensemble Regularization) to optimize coolingprocedures for Google’s data centers.
Figure 2.8.3 presents the energy-saving results from one particular BCOOLER experiment. At theend of the three-month experiment, BCOOLER achieved roughly 12.7% energy savings. BCOOLER
was able to achieve these savings while maintaining the cooling comfort levels that the building
managers preferred.
Energy Savings Results Over Time for Select BCOOLER Experiment
Source: Luo et al., 2022 | Chart: 2023 AI Index Report

12.7%
12%

Cumulative AI Savings (%)

10%

8%

6%

4%

2%

0%

2021-Aug-01

2021-Aug-15

2021-Aug-29

2021-Sep-12

2021-Sep-26

2021-Oct-10

2021-Oct-24

Figure 2.8.3

Chapter 2 Preview

54Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.9 AI for Science

2022 was a groundbreaking year for AI in science. This subsection looks at some meaningful ways in which AI has recently been used
to accelerate scientific discovery.

2.9 AI for Science
Accelerating Fusion Science
Through Learned Plasma Control

Photos of the Variable Configuration Tokamak (TCV) at EPFL
Source: DeepMind, 2022

Nuclear fusion could generate clean
energy by fusing hydrogen. A common
approach to achieving nuclear fusion
is using a tokamak, a machine which
controls and contains the heatedhydrogen plasma (Figure 2.9.1). However,the plasmas produced in these machines
are unstable and necessitate constantFigure 2.9.1

monitoring. In 2022, researchers at
DeepMind developed a reinforcement
learning algorithm to discover optimal
tokamak management procedures.

Discovering Novel Algorithms
for Matrix Manipulation With
AlphaTensor

A Demonstration of AlphaTensor’s Matrix Manipulation Process
Source: Fawzi et al., 2022

Matrix multiplication is a simple algebraic
operation that is essential to many
computations, including neural networks
and scientific computing (Figure 2.9.2).
The classic algorithm to multiply two 2x2

Figure 2.9.2

matrices takes 2^3 = 8 multiplications.
Strassen discovered 50 years ago
how to reduce this to 7, and generally
how to multiply two n x n matrices inincluding 4x4 matrices over the integers [0,1]. It also matches state-O(n^ log(7)) operations. DeepMind’sof-the-art performance on several other matrix sizes, including 4x4

AlphaTensor uses Reinforcement

over the integers. It does this by searching through large numbers

Learning to improve on state-of-the-of possible algorithms, and evaluating them over real computer

art algorithms for many matrix sizes,

architectures.

Chapter 2 Preview

55Artificial IntelligenceIndex Report 2023

Designing Arithmetic Circuits With
Deep Reinforcement LearningThis year, a team at Nvidia discovered a

Chapter 2: Technical Performance
2.9 AI for Science

A Juxtaposition of Nvidia Circuits Designed by
PrefixRL Vs. EDA Tools
Source: Roy et al., 2022

novel approach to improving the chipsthat power AI systems: Use AI systems todesign better chips. They were able to train
a reinforcement learning agent to design
chip circuits that are smaller, faster, andmore efficient than the circuits designed byelectronic design automation tools (EDAs).
One of Nvidia’s latest categories of chips,
the Hopper GPU architecture, has over
13,000 instances of AI-designed circuits.

Figure 2.9.3

Figure 2.9.3 shows a 64-bit adder circuitdesigned by Nvidia’s PrefixRL AI agent(on the left) which is 25% smaller while
being just as fast and functional as those
designed by the state-of-the-art EDA tools.

Unlocking de Novo Antibody
Design With Generative AI

Zero-Shot Generative AI for de Novo Antibody Design
Source: Shanehsazzadeh et al., 2023

Antibody discovery, which is referred toas de novo antibody discovery, typicallyrequires immense amounts of time andresources. Traditional methods for denovo discovery offer little control over
the outputs, so that proposed antibodies
are often suboptimal. To that end, a teamof researchers turned to generative AI
models to create antibodies in a zero-shot
fashion, where antibodies are created with
one round of model generation without

Figure 2.9.4

further optimizations (Figure 2.9.4). These
AI-generated antibodies are also robust.
The fact that generative AI can create newantibodies has the potential to accelerate
drug discovery.

Chapter 2 Preview

56Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Appendix
ImageNet

Very Deep Convolutional Networks for Large-Scale
Image Recognition

Data on ImageNet accuracy was retrieved through a

ViTAEv2: Vision Transformer Advanced by Exploring

detailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.

Inductive Bias for Image Recognition and Beyond

The reported dates correspond to the year in which

To highlight progress on top-1 accuracy with the use

a paper was first published to arXiv, and the reportedof extra training data, scores were taken from theresults (top-1 accuracy) correspond to the result

following papers:

reported in the most recent version of each paper.

Big Transfer (BiT): General Visual
Representation Learning

Learn more about the LSVRC ImageNet competition
and the ImageNet dataset.
To highlight progress on top-1 accuracy without theuse of extra training data, scores were taken from thefollowing papers:
Aggregated Residual Transformations for
Deep Neural Networks
Exploring the Limits of Weakly Supervised Pretraining
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
ImageNet Classification With Deep Convolutional
Neural Networks
PeCo: Perceptual Codebook for BERT
Pre-training of Vision Transformers
Progressive Neural Architecture Search

CoAtNet: Marrying Convolution and
Attention for All Data Sizes
CoCa: Contrastive Captioners Are Image-Text
Foundation Models
Meta Pseudo Labels

National Institute ofStandards and Technology(NIST) Face Recognition
Vendor Test (FRVT)
Data on NIST FRVT 1:1 verification accuracy bydataset was obtained from the FRVT 1:1 verification
leaderboard.

Rethinking the Inception Architecture for
Computer Vision
Self-Training With Noisy Student Improves
ImageNet Classification
Some Improvements on Deep Convolutional Neural
Network Based Image Classification

Chapter 2 Preview

57Artificial IntelligenceIndex Report 2023

Celeb-DF
Data on Celeb-DF AUC was retrieved through adetailed arXiv literature review. The reported dates
correspond to the year in which a paper was first

Appendix
Chapter 2: Technical Performanceversion of each paper. Details on the MPII benchmark
can be found in the MPII paper and MPII dataset.
To highlight progress on percentage of correct
keypoints without the use of extra training data, scores

published to arXiv or a method was introduced. With

were taken from the following papers:

Celeb-DF, recent researchers have tested previously

Bottom-Up and Top-Down Reasoning
With Hierarchical Rectified Gaussians

existing deepfake detection methodologies. The year
in which a method was introduced, even if it was
subsequently tested, is the year in which it is includedin the report. The reported results (AUC) correspondto the result reported in the most recent version ofeach paper. Details on the Celeb-DF benchmark can
be found in the Celeb-DF paper.
To highlight progress on Celeb-DF, scores were takenfrom the following papers:
Deepfake Detection via Joint Unsupervised
Reconstruction and Supervised Classification
Exposing Deepfake Videos by Detecting
Face Warping Artifacts
Face X-Ray for More General Face Forgery Detection
FaceForensics++: Learning to Detect Manipulated
Facial Images
Spatial-Phase Shallow Learning: Rethinking Face
Forgery Detection in Frequency Domain

Cascade Feature Aggregation for
Human Pose Estimation
Deeply Learned Compositional Models for
Human Pose Estimation
Efficient Object Localization Using
Convolutional Networks
Learning Feature Pyramids for Human Pose Estimation
Stacked Hourglass Networks for
Human Pose Estimation
Toward Fast and Accurate Human Pose Estimation
via Soft-Gated Skip Connections
ViTPose: Simple Vision Transformer Baselines for
Human Pose Estimation

Cityscapes Challenge,
Pixel-Level Semantic
Labeling Task

MPII

Data on the Cityscapes challenge, pixel-level semantic

Data on MPII percentage of correct keypoints (PCK)

was taken from the Cityscapes dataset, specificallywas retrieved through a detailed arXiv literature

their pixel-level semantic labeling leaderboard.

review cross-referenced by technical progress

More details about the Cityscapes dataset and otherreported on Papers With Code. The reported dates

corresponding semantic segmentation challenges cancorrespond to the year in which a paper was firstbe accessed at the Cityscapes dataset webpage.

labeling task mean intersection-over-union (mIoU)

published to arXiv, and the reported results (PCK)
correspond to the result reported in the most recent

Chapter 2 Preview

58Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Kvasir-SEG

An Analysis of Scale Invariance in Object
Detection-SNIP

Data on Kvasir-SEG mean dice was retrieved through

CBNet: A Novel Composite Backbone Network
Architecture for Object Detection

a detailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.
The reported dates correspond to the year in whicha paper was first published to arXiv, and the reported
results (mean dice) correspond to the result reported
in the most recent version of each paper. Details on

Deformable ConvNets v2: More Deformable,
Better Results
DetectoRS: Detecting Objects With Recursive
Feature Pyramid and Switchable Atrous Convolution

the Kvasir-SEG benchmark can be found in the Kvasir-

EVA: Exploring the Limits of Masked Visual
Representation Learning at Scale

SEG paper.

Grounded Language-Image Pre-training

To highlight progress on Kvasir-SEG, scores were

Inside-Outside Net: Detecting Objects in Context
With Skip Pooling and Recurrent Neural Networks

taken from the following papers:
GMSRF-Net: An Improved Generalizability With
Global Multi-Scale Residual Fusion Network for
Polyp Segmentation

CIFAR-10

PraNet: Parallel Reverse Attention Network for
Polyp Segmentation

a detailed arXiv literature review cross-referenced by

ResUNet++: An Advanced Architecture for
Medical Image Segmentation

reported dates correspond to the year in which a paper

Spatially Exclusive Pasting: A General Data
Augmentation for the Polyp Segmentation(FID score) correspond to the result reported in the most

Common Object in Context(COCO)
Data on COCO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review
cross-referenced by technical progress reported on
Papers With Code. The reported dates correspond to
the year in which a paper was first published to arXiv,

Data on CIFAR-10 FID scores was retrieved throughtechnical progress reported on Papers With Code. The
was first published to arXiv, and the reported resultsrecent version of each paper. Details on the CIFAR-10
benchmark can be found in the CIFAR-10 paper.
To highlight progress on CIFAR-10, scores were takenfrom the following papers:
GANs Trained by a Two Time-Scale Update Rule
Converge to a Local Nash Equilibrium
Large Scale GAN Training for High Fidelity Natural
Image Synthesis

and the reported results (mAP50) correspond to theRefining Generative Process With Discriminator
Guidance in Score-Based Diffusion Models

result reported in the most recent version of each

Score-Based Generative Modeling in Latent Space

paper. Details on the COCO benchmark can be foundin the COCO paper.
To highlight progress on COCO, scores were takenfrom the following papers:
Chapter 2 Preview

Score-Based Generative Modeling Through
Stochastic Differential Equations
Self-Supervised GAN: Analysis and Improvement
With Multi-Class Minimax Game59

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

STL-10

The reported dates correspond to the year in which

Data on STL-10 FID scores was retrieved through a

results (accuracy) correspond to the result reported indetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code. The
reported dates correspond to the year in which a paper
was first published to arXiv, and the reported results(FID score) correspond to the result reported in themost recent version of each paper. Details on the STL-a paper was first published to arXiv, and the reported
the most recent version of each paper. Human-level
performance is taken from the 2021 VQA challenge.
To highlight progress on VQA accuracy without theuse of extra training data, scores were taken from thefollowing papers:

10 benchmark can be found in the STL-10 paper.

Bilinear Attention Networks

To highlight progress on STL-10, scores were taken

Multimodal Compact Bilinear Pooling for Visual
Question Answering and Visual Grounding

from the following papers:
DEGAS: Differentiable Efficient Generator Search
Diffusion-GAN: Training GANs With Diffusion
Discriminator Contrastive Divergence:
Semi-Amortized Generative Modeling by
Exploring Energy of the Discriminator
Dist-GAN: An Improved GAN Using
Distance Constraints
Soft Truncation: A Universal Training Technique of
Score-Based Diffusion Model for High Precision
Score Estimation

Text-to-Image Models
on MS-COCO 256 × 256
FID-30K
Data on MS-COCO 256 x 256 FID 30K for Text-toImage Models was retrieved from the paper Saharia
et al., 2022.

Oscar: Object-Semantics Aligned Pre-training
for Vision-Language Tasks
PaLI: A Jointly-Scaled Multilingual
Language-Image Model
Tips and Tricks for Visual Question Answering:
Learnings From the 2017 Challenge
UNITER: UNiversal Image-TExt Representation Learning
VLMo: Unified Vision-Language Pre-training With
Mixture-of-Modality-Experts

BEiT-3 Vs. Previous SOTA
Data on BEiT-3 and Previous SOTA was retrieved fromthe paper Wang et al., 2022.

Visual Commonsense
Reasoning (VCR)
Data on VCR Q->AR score was taken from VCR
leaderboard; the VCR leaderboard webpage further

Visual Question Answering(VQA)

delineates the methodology behind the VCR challenge.

Data on VQA accuracy was retrieved through ain the VCR paper.

Human performance on VCR is taken from Zellers etal., 2018. Details on the VCR benchmark can be founddetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.

Chapter 2 Preview

60Artificial IntelligenceIndex Report 2023

Kinetics-400, Kinetics-600,
and Kinetics-700
Data on Kinetics-400, Kinetics-600, and Kinetics-700
accuracy was retrieved through a detailed arXiv

Appendix
Chapter 2: Technical Performance

To highlight progress on Kinetics-600, scores weretaken from the following papers:
Learning Spatio-Temporal Representation
With Local and Global Diffusion

literature review cross-referenced by technical

Masked Feature Prediction for Self-Supervised
Visual Pre-training

progress reported on Papers With Code (Kinetics-400,

PERF-Net: Pose Empowered RGB-Flow Net

Kinetics-600, and Kinetics-700). The reported

Rethinking Spatiotemporal Feature Learning:
Speed-Accuracy Trade-Offs in Video Classification

dates correspond to the year in which a paper was
first published to arXiv, and the reported results(top-1 accuracy) correspond to the result reported

Rethinking Video ViTs: Sparse Video Tubes for
Joint Image and Video Learningin the most recent version of each paper. Details on

SlowFast Networks for Video Recognition

the Kinetics-400 benchmark can be found in the
Kinetics-400 paper. Details on the Kinetics-600
benchmark can be found in the Kinetics-600 paper.
Details on the Kinetics-700 benchmark can be found in
the Kinetics-700 paper.
To highlight progress on Kinetics-400, scores weretaken from the following papers:
Co-training Transformer With Videos and Images
Improves Action Recognition
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Large-Scale Weakly-Supervised Pre-training forVideo Action Recognition
Non-Local Neural Networks
Omni-Sourced Webly-Supervised Learning forVideo Recognition

To highlight progress on Kinetics-700, scores weretaken from the following papers:
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Learn to Cycle: Time-Consistent Feature Discovery
for Action Recognition
Masked Feature Prediction for Self-Supervised
Visual Pre-training

Text-to-Video Models
on UCF-101
Data on UCF-101 Inception Score (IS) for Text-to-Video
Models was retrieved from the paper Hong et al., 2022,
and Singer et al., 2022.

SlowFast Networks for Video Recognition
Temporal Segment Networks: Towards Good
Practices for Deep Action Recognition

Chapter 2 Preview

61Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical PerformanceSuperGLUEThe SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about theSuperGLUE benchmark are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks andevaluation metrics for SuperGLUE are:NAME

IDENTIFIERMETRIC

Broadcoverage Diagnostics

AX-b

Matthew’s Carr

CommitmentBank

CB

Avg. F1/Accuracy

Choice of Plausible Alternatives

COPA

Accuracy

Multi-Sentence Reading Comprehension

MultiRC

F1a/EM

Recognizing Textual Entailment

RTE

Accuracy

Words in Context

WiC

Accuracy

The Winograd Schema Challenge

WSC

Accuracy

BooIQ

BooIQ

Accuracy

Reading Comprehension With Commonsense Reasoning

ReCoRD

F1/Accuracy

Winogender Schema Diagnostic

AX-g

Gender Parity/Accuracy

Reading Comprehension Dataset Requiring
Logical Reasoning (ReClor)
Data on ReClor accuracy was retrieved through the ReClor leaderboard cross-referenced by technical progress
reported on Papers With Code. Details about the ReClor benchmark can be found in the ReClor paper.

Chapter 2 Preview

62Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Narrative Highlight:
Just How Much Better Have
Language Models Become?

paid $90,000 for the role, and before I even started

Complete answer outputted by GPT-2 to the AI Index

I wanted to know in advance: What percentage ofprompt request.

his work was actually done as a private citizen, andModel prompt >>> Explain to me the major

how efficiently should he handle it for his clients? And

accomplishments of Theodore Roosevelt’s

what advice would he give me should I end up in thepresidency.

public eye?

(The most notable is the establishment of a national

At no time, my job as the lead prosecutor on thistelephone and mail exchange company.)case required me to put forth any of those two or

But the only things that will change with a president
who knows how to operate a business is his
understanding of the American system of government.
The public knows all about our problems, yet at least
half of the people who need it won’t know anything

on the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with my leadprosecutor.

five requests. Most people on the case, though, did.
One person asked me what the case’s real value
was. That didn’t stop me from telling it to one of
my top managers. And my top manager never evenmentioned it.

about ours. We can no longer have this. Our system ofIn the end, it was a two-week, $40 million round-government is the only

trip tour of the government’s business, with the best

one in which you really need a public relations firm towork hard.
The system of governmental control, while working,is not always effective in ensuring that the very things

lawyer possible. In exchange, we got a chance to hear
his last public comment in this case, his last comment
to me.
There was one more point — the end point of the case.

that our economy needs, do not always be at the

When it came up for consideration in October of 1993,

top of everybody’s list. So I wanted to know how

I had the choice to give it three years, or six. And I

successful the system you employ for managing thechose six.

public’s needs is in helping the American economy
grow, how efficient the service our economy provides
is being made, and how it can better serve the needsof a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer to work on
the case. This was before I ever hired my own law firm
to handle government investigations. For my first majorjob, at least, I had to convince two top aides to get

Chapter 2 Preview

63Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Large Language Models on
the Blocksworld Domain

PubMedData on Blocksworld domain for large language models

evaluation (ROUGE-1) was retrieved through a detailed

Data on PubMed recall-oriented understudy for gisting

was retrieved from the paper Valmeekam et al., 2022.

arXiv literature review cross-referenced by technicalarXivdates correspond to the year in which a paper was first

Data on arXiv recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers With Code. The reported
dates correspond to the year in which a paper was firstprogress reported on Papers With Code. The reported
published to arXiv, and the reported results (ROUGE-1)
correspond to the result reported in the most recent
version of each paper. Details about the PubMedbenchmark are in the PubMed paper.
To highlight progress in PubMed, scores were taken

published to arXiv, and the reported results (ROUGE-1)

from the following papers:

correspond to the result reported in the most recent

A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents

version of each paper. Details about the arXiv
benchmark are in the arXiv dataset webpage.
To highlight progress on arXiv, scores were taken fromthe following papers:
Big Bird: Transformers for Longer Sequences
A Discourse-Aware Attention Model for
Abstractive Summarization of Long Documents
Get to the Point: Summarization With
Pointer-Generator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
MemSum: Extractive Summarization of Long
Documents Using Multi-Step Episodic Markov
Decision Processes
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization

Get to the Point: Summarization With PointerGenerator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
LongT5: Efficient Text-to-Text Transformer forLong Sequences
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization
Sparsifying Transformer Models With Trainable
Representation Pooling

Abductive Natural Language
Inference (aNLI)
Data on Abductive Natural Language Inference (aNLI)
was sourced from the Allen Institute for AI’s aNLI
leaderboard. Details on the aNLI benchmark can befound in the aNLI paper.

Chapter 2 Preview

64Artificial IntelligenceIndex Report 2023

SST-5 Fine-Grained
Data on SST-5 Fine-Grained accuracy was retrieved
through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
With Code. The reported dates correspond to the year

Appendix
Chapter 2: Technical Performance

To highlight progress on MMLU accuracy, scores weretaken from the following papers:
Language Models Are Few-Shot Learners
Language Models Are Unsupervised
Multitask Learners

in which a paper was first published to arXiv, and theScaling Instruction-Finetuned Language Models

reported results (accuracy) correspond to the result

Scaling Language Models: Methods, Analysis &Insights from Training Gopher

reported in the most recent version of each paper.
Details about the SST-5 Fine-Grained benchmark can
be found in the SST paper.
To highlight progress on SST-5 Fine-Grained accuracy,
scores were taken from the following papers:
An Algorithm for Routing Capsules in All Domains
An Algorithm for Routing Vectors in Sequences
Improved Semantic Representations from TreeStructured Long Short-Term Memory Networks

Number of Commercially
Available MT Systems
Details about the number of commercially available
MT systems were sourced from the Intento report The
State of Machine Translation, 2022. Intento is a San
Francisco–based startup that analyzes commerciallyavailable MT services.

Improved Sentence Modeling Using Suffix
Bidirectional LSTM

VoxCeleb

Learned in Translation: Contextualized Word Vectors

Data on VoxCeleb equal error rate (EER) was retrieved

Less Grammar, More Features
Recursive Deep Models for Semantic Compositionality
Over a Sentiment Treebank
Self-Explaining Structures Improve NLP Models

MMLU
Data on MMLU accuracy was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code. The
reported dates correspond to the year in which a paper
was first published to arXiv, and the reported results(accuracy) correspond to the result reported in the mostrecent version of each paper. Details about the MMLU
benchmark can be found in the MMLU paper.

from the VoxCeleb Speaker Recognition Challenge(VoxSRC).
For the sake of consistency, the AI Index reported scoreson the initial VoxCeleb dataset. Specifically, the AI Index
made use of the following sources of information:
ID R&D System Description to VoxCeleb Speaker
Recognition Challenge 2022
The IDLAB VoXSRC-20 Submission: Large MarginFine-Tuning and Quality-Aware Score Calibration in
DNN Based Speaker Verification
The SpeakIn System for VoxCeleb Speaker
Recognition Challenge 2021
VoxCeleb: A Large-Scale Speaker Identification
Dataset
VoxCeleb: Large-Scale Speaker Verification in the Wild
VoxCeleb2: Deep Speaker Recognition

Chapter 2 Preview

65Artificial IntelligenceIndex Report 2023

Whisper
Data on Whisper for large-scale speech recognition
models was retrieved from the paper Radford et al.,2022.

Procgen
Data on Procgen mean-normalized score was retrieved
through a detailed arXiv literature review. The reported
dates correspond to the year in which a paper was firstpublished to arXiv, and the reported results (meannormalized score) correspond to the result reported in
the most recent version of each paper. Details on the
Procgen benchmark can be found in the Procgen paper.

Appendix
Chapter 2: Technical Performance

Training Time, Numberof Accelerators, andPerformance
Data on training time, number of accelerators,
and performance for AI systems was taken from
the MLPerf Training and Inference benchmark
competitions. Details on the MLPerf Training
benchmark can be found in the MLPerf Training
Benchmark paper, while details on MLPerf Inference
can be found in the MLPerf Inference Benchmark
paper. Information about the current benchmark
categories as well as technical information about
submission and competition subdivisions can be

To highlight progress on Procgen, scores were taken

found on the MLPerf Training and MLPerf Inference

from the following papers:

webpages.

Automatic Data Augmentation for Generalization in
Reinforcement Learning

The AI Index made use of data from the following

Leveraging Procedural Generation to Benchmark
Reinforcement Learning

MLPerf Training v2.1, 2022

Procedural Generalization by Planning With
Self-Supervised World Models

MLPerf Training v1.1, 2021

Rethinking Value Function Learning for
Generalization in Reinforcement Learning

MLPerf Training competitions:
MLPerf Training v2.0, 2022
MLPerf Training v1.0, 2021
MLPerf Training v0.7, 2020
MLPerf Training v0.6, 2019
MLPerf Training v0.5, 2018
The AI Index made use of data from the following
MLPerf Inference competitions:
MLPerf Inference v2.1, 2022
MLPerf Inference v2.0, 2022
MLPerf Inference v1.1, 2021
MLPerf Inference v1.0, 2021
MLPerf Inference v0.7, 2020

Chapter 2 Preview

66