Artificial IntelligenceIndex Report 2022

CHAPTER 2:

Technical
Performance

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER 2:

Chapter Preview
Overview

4

Chapter Highlights

5

2.1 COMPUTER VISION—IMAGE

6Image Classification

6

ImageNet

6

ImageNet: Top-1 Accuracy

6

ImageNet: Top-5 Accuracy

6

Image Generation

8

	STL-10: Fréchet Inception
Distance (FID) Score

8

CIFAR-10: Fréchet Inception
Distance (FID) Score

9

Medical Image Segmentation

15

CVC-ClinicDB and Kvasir-SEG

15

Face Detection and Recognition

16

National Institute of Standards
and Technology (NIST) Face
Recognition Vendor Test (FRVT)

16

Face Detection: Effects of Mask-Wearing

17

Face Recognition Vendor Test (FRVT):
Face-Mask Effects

17

Highlight: Masked Labeled Faces
in the Wild (MLFW)

18

Visual Reasoning

19

Visual Question Answering(VQA) Challenge

19

Deepfake Detection

10

FaceForensics++

10

2.2 COMPUTER VISION—VIDEO

21

Celeb-DF

11

Activity Recognition

21

Human Pose Estimation

11

Kinetics-400, Kinetics-600, Kinetics-700 21

Leeds Sports Poses: Percentage
of Correct Keypoints (PCK)

12

ActivityNet: Temporal Action
Localization Task

Human3.6M: Average Mean
Per Joint Position Error (MPJPE)

13

Semantic Segmentation
Cityscapes

Object Detection
14
14

23
24

Common Object in Context (COCO)

25

You Only Look Once (YOLO)

26

Visual Commonsense Reasoning (VCR)

27

ACCESS THE PUBLIC DATA

2

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER 2: CHAPTER PREVIEW (CONT’D)
2.3 LANGUAGE

28

2.4 SPEECH

40

English Language Understanding

28

Speech Recognition

40

SuperGLUE

28

Transcribe Speech: LibriSpeech(Test-Clean and Other Datasets)

40

VoxCeleb

41

Stanford Question Answering
Dataset (SQuAD)

29

Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)30

2.5 RECOMMENDATION

42

32

42

Text Summarization
arXiv

32

Commercial Recommendation:
MovieLens 20M

PubMed33

Click-Through Rate Prediction: Criteo

43

2.6 REINFORCEMENT LEARNING

44

Reinforcement Learning Environments

44

Natural Language Inference
Stanford Natural Language
Inference (SNLI)
Abductive Natural Language
Inference (aNLI)
Sentiment Analysis
SemEval 2014 Task 4 Sub Task 2
Machine Translation (MT)34
34

Arcade Learning Environment: Atari-57 44
35

Procgen45

36

Human Games: Chess

47

36
37

WMT 2014, English-German
and English-French

38

Number of Commercially
Available MT Systems

39

ACCESS THE PUBLIC DATA

2.7 HARDWARE

48

MLPerf: Training Time48

MLPerf: Number of Accelerators

50

IMAGENET: Training Cost

51

2.8 ROBOTICS

52

Price Trends in Robotic Arms

52

AI Skills Employed by Robotics Professors 53
APPENDIX

54

3

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE

Overview
This year, the technical performance chapter includes more analysis
than ever before of the technical progress in various subfields ofartificial intelligence, including trends in computer vision, language,
speech, recommendation, reinforcement learning, hardware, androbotics. It uses a number of quantitative measurements, from
common AI benchmarks and prize challenges to a field-wide survey,
to highlight the development of top-performing AI systems.

Chapter 2 Preview

4Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER HIGHLIGHTS• Data, data, data: Top results across technical benchmarks have increasingly relied on the use of
extra training data to set new state-of-the-art results. As of 2021, 9 state-of-the-art AI systems
out of the 10 benchmarks in this report are trained with extra data. This trend implicitly favors
private sector actors with access to vast datasets.
• Rising interest in particular computer vision subtasks: In 2021, the research community saw
a greater level of interest in more specific computer vision subtasks, such as medical image
segmentation and masked-face identification. For example, only 3 research papers tested
systems against the Kvasir-SEG medical imaging benchmark prior to 2020. In 2021, 25
research papers did. Such an increase suggests that AI research is moving toward research that
can have more direct, real-world applications.
• AI has not mastered complex language tasks, yet: AI already exceeds human performance levels
on basic reading comprehension benchmarks like SuperGLUE and SQuAD by 1%–5%. Although
AI systems are still unable to achieve human performance on more complex linguistic tasks
such as abductive natural language inference (aNLI), the difference is narrowing. Humans
performed 9 percentage points better on aNLI in 2019. As of 2021, that gap has shrunk to 1.
• Turn toward more general reinforcement learning: For the last decade, AI systems havebeen able to master narrow reinforcement learning tasks in which they are asked to maximize
performance in a specific skill, such as chess. The top chess software engine now exceeds
Magnus Carlsen’s top ELO score by 24%. However, in the last two years AI systems have also
improved by 129% on more general reinforcement learning tasks (Procgen) in which theymust operate in novel environments. This trend speaks to the future development of AI systems
that can learn to think more broadly.
• AI becomes more affordable and higher performing: Since 2018, the cost to train an image
classification system has decreased by 63.6%, while training times have improved by 94.4%.
The trend of lower training cost but faster training time appears across other MLPerf task
categories such as recommendation, object detection and language processing, and favors themore widespread commercial adoption of AI technologies.
• Robotic arms are becoming cheaper: An AI Index survey shows that the median price ofrobotic arms has decreased by 46.2% in the past five years—from $42,000 per arm in 2017 to$22,600 in 2021. Robotics research has become more accessible and affordable.

Chapter 2 Preview

5Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Computer vision is the subfield of AI that teaches machines to understand images and videos. There is a wide range of computer
vision tasks, such as image classification, object recognition, semantic segmentation, and face detection. As of 2021, computers canoutperform humans on a plethora of computer vision tasks. Computer vision technologies have a variety of important real-world
applications, such as autonomous driving, crowd surveillance, sports analytics, and video-game creation.

2.1 COMPUTER VISION—IMAGEI M AG E C L A S S I F I CAT I O N​Image classification refers to the ability of machines tocategorize what they see in images (Figure 2.1.1). In apractical sense, image recognition systems can help cars
identify objects in their surroundings, doctors detecttumors, and factory managers spot production defects.
The past decade has seen tremendous advances in thetechnical capacity of image recognition systems, especiallyas researchers have embraced more machine learningtechniques. Moreover, progress in algorithmic, hardware,and data technologies has meant that image recognition
has become more affordable, widely applicable, andaccessible than ever before.

A DEMONSTRATION OF IMAGE CLASSIFICATION
Source: Krizhevsky, 2020

ImageNet
ImageNet is a database that includes over 14 million images
across 20,000 categories publicly available to researchers
working on image classification problems. Created in 2009,
ImageNet is now one of the most common ways scientists
benchmark improvement on image classification.

ImageNet: Top-1 Accuracy
Benchmarking on ImageNet is measured through accuracy
metrics, which quantify how frequently AI systems assign
the right labels to the given images. Top-1 accuracy
measures the rate at which the top prediction made bya classification model for a given image matches theimage’s actual target label. In recent years, it has become
increasingly common to improve the performance ofsystems on ImageNet by pretraining them with additional
data from other image datasets.
As of late 2021, the top image classification system makes
on average 1 error for every 10 classification attempts
on Top-1 accuracy compared to an average of 4 errors
for every 10 attempts in late 2012 (Figure 2.1.2). In 2021,

Chapter 2 Preview

Figure 2.1.1

the top pretrained system was CoAtNets, produced byresearchers on the Google Brain Team.

ImageNet: Top-5 Accuracy
Top-5 accuracy considers whether any of the model’s 5highest probability answers align with the image label. As
highlighted in Figure 2.1.3, AI systems presently achieve
near perfect Top-5 estimation. Currently, the state-of-theart performance on Top-5 accuracy with pretraining is
99.0%, achieved in November 2021 by Microsoft Cloud andMicrosoft AI’s Florence-CoSwim-H model.
Improvements in Top-5 accuracy on ImageNet seem to beplateauing, which is perhaps unsurprising. If your system
is classifying correctly 98 or 99 out of 100 times, there is
only so much higher you can go.

6Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

IMAGENET CHALLENGE: TOP-1 ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

90.88%, With Extra Training Data

90%

Top-1 Accuracy (%)

87.80%, Without Extra Training Data

80%

70%

60%
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.2

IMAGENET CHALLENGE: TOP-5 ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

100%

99.02%, With Extra Training Data

97.90%, Without Extra Training Data

Top-5 Accuracy (%)

95%

94.90 %, Human Baseline

90%

85%

80%
2012

Chapter 2 Preview

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.3

7

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageI M AG E G E N E R AT I O NImage generation is the task of generating images that are
indistinguishable from real ones. Image generation can be
widely useful in generative domains where visual content
has to be created, for example entertainment (companies
like NVIDIA have already used image generators to create
virtual worlds for gaming), fashion (designers can let AI

systems generate different design patterns), and healthcare(image generators can synthetically create novel drug
compounds). Figure 2.1.4 illustrates progress made in
image generation by presenting several human faces thatwere synthetically generated by AI systems in the last year.

GAN PROGRESS ON FACE GENERATION
Source: Goodfellow et al., 2014; Radford et al., 2016; Liu & Tuzel, 2016;
Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al.,2020; AI Index, 2021; Vahdat et al., 2021

2021

STL-10: Fréchet Inception Distance (FID) Score
The Fréchet Inception Distance score tracks the similarity
between an artificially generated set of images and thereal images from which it was generated. A low scoremeans that the generated images are more similar to
the real ones, and a score of zero indicates that the fakeimages are identical to the real ones.

Figure 2.1.4

Figure 2.1.5 documents the gains generative models havemade in FID on the STL-10 dataset, one of the most widely
cited datasets in computer vision. The state-of-the-art
model on STL-10 developed by researchers at the Korea
Advanced Institute of Science and Technology as well as the
University of Seoul posted a FID score of 7.7, significantly
better than the state-of-the-art result from 2020.

STL-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE

Fréchet Inception Distance (FID) Score

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

30

20

10
7.71

0
2018

2019

2020

2021
Figure 2.1.5

Chapter 2 Preview

8Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageCIFAR-10: Fréchet Inception Distance (FID) Score
Progress on image generation can also be benchmarked
on CIFAR-10, a dataset of 60,000 color images across 10
different object classes. The state-of-the-art results on
CIFAR-10 posted in 2021 were achieved by researchers
from NVIDIA.

The FID scores achieved by the top image generation
models are much lower on CIFAR-10 than STL-10. Thisdifference is likely attributable to the fact that CIFAR-10
contains images of much lower resolution (32 x 32 pixels)
than those on STL-10 (96 x 96 pixels).

CIFAR-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Fréchet Inception Distance (FID) Score

30

20

10

2.10

0
2017

2018

2019

2020

2021
Figure 2.1.6

Chapter 2 Preview

9Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

D E E P FA K E D E T E CT I O N

dataset that contains approximately 1,000 original video
sequences sourced from YouTube videos. Progress on
FaceForensics++ is measured in terms of accuracy: thepercentage of altered images an algorithm can correctly
identify.

Many AI systems can now generate fake images that are
indistinguishable from real ones. A related technology
involves superimposing one person’s face onto another,creating a so-called “deepfake.” Deepfakes are used
for purposes ranging from advertising to generating
misogynistic pornography and disinformation (in 2018,
for example, a deepfake video of Barack Obama uttering
profanities about Donald Trump was circulated online over2 million times). In the last few years, AI researchers havesought to keep up with improving deepfake technologies by
crafting stronger deepfake detection algorithms.

Although FaceForensics++ was introduced in 2019,researchers have tested previously existing deepfake
detection methods on the dataset in order to trackprogress over time in deepfake detection (Figure 2.1.7).
In the last decade, AI systems have become better andbetter at detecting deepfakes. In 2012, the top-performing
systems could correctly identify 69.9% of deepfakes
across all four FaceForenics++ datasets. In 2021, that
number increased to 97.7%.1

FaceForensics++
FaceForensics++ is a deepfake detection benchmarking

FACEFORENSICS++: ACCURACY
Source: arXiv, 2021 | Chart: 2022 AI Index Report

99.98%, Face2Face
99.47%, DeepFake
98.27%, FaceSwap

100%

93.25%, NeuralTextures

Accuracy (%)

90%

80%

70%

60%
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.7

1 These numbers were taken by averaging performance across all four FaceForensics++ datasets.

Chapter 2 Preview

10Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Celeb-DF
The Celeb-DF deepfake detection dataset is composed of
590 original celebrity YouTube videos manipulated into
5,639 deepfakes. Celeb-DF was introduced in 2019. In
2021, the top score on Celeb-DF was 76.9 and came fromresearchers at the University of Science and Technology ofChina and Alibaba group (Figure 2.1.8).

Top detection models perform significantly worse (by 20
percentage points) on Celeb-DF than FaceForensics++,suggesting that Celeb-DF is a more challenging dataset totest out techniques on. As deepfake technologies continue
to improve in the upcoming years, it will be important
to continue monitoring progress on Celeb-DF and othersimilarly challenging deepfake detection datasets.

CELEB-DF: AREA UNDER CURVE SCORE (AUC)
Source: arXiv, 2021 | Chart: 2022 AI Index Report80

Area Under Curve Score (AUC)

76.88

75

70

65

60
2018

2019

2020

2021
Figure 2.1.8

H U M A N P O S E E S T I M AT I O N
Human pose estimation is the task of estimating different
positions of human body joints (arms, head, torso, etc.)from a single image (Figure 2.1.9), and then combining theseestimates to correctly label the pose the human is taking.

Human pose estimation can be used to facilitate activity
recognition for purposes such as sports analytics, crowd
surveillance, CGI development, virtual environment design,and transportation (for example, identifying the bodylanguage signs of an airport runway controller).

A DEMONSTRATION OF
HUMAN POSE ESTIMATION
Source: Cao et al., 2019

Figure 2.1.9

Chapter 2 Preview

11Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Leeds Sports Poses: Percentage of Correct
Keypoints (PCK)In 2021, the top-performing human pose estimation
model correctly identified 99.5% of keypoints on Leeds
Sports Poses (Figure 2.1.10). Given that maximum
performance on Leeds Sports is 100.0%, more challenging
benchmarks for human pose estimation will have to be
developed, as we are close to saturating the benchmark.

The Leeds Sports Poses dataset contains 2,000 images
collected from Flickr of athletes playing a sport. Each
image includes information on 14 different body joint
locations. Performance on the Leeds Sports Poses
benchmark is assessed by the percentage of correctly
estimated keypoints.

LEEDS SPORTS POSES: PERCENTAGE of CORRECT KEYPOINTS (PCK)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

99.50%

Percentage of Correct Keypoints (PCK)

100%

90%

80%

70%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.10

Chapter 2 Preview

12Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Human3.6M: Average Mean Per Joint Position
Error (MPJPE)
3D human pose estimation is a more challenging type ofpose estimation, where AI systems are asked to estimate
poses in a three- rather than two-dimensional space. The
Human3.6M dataset tracks progress in 3D human pose
estimation. Human3.6M is a collection of over 3.6 million
images of 17 different types of human poses (talking on
the phone, discussing, and smoking, etc.). Performanceon Human3.6M is measured in average mean per joint
position error in millimeters, which is the average difference
between an AI model’s position estimations and the actual
position annotation.
In 2014, the top-performing model was making an average
per joint error of 16 centimeters, half the size of a standard
school ruler. In 2021, this number fell to 1.9 centimeters,less than the size of an average paper clip.

HUMAN3.6M: AVERAGE MEAN PER JOINT POSITION ERROR (MPJPE)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

150

Average MPJPE (mm)

120

90

60

22.70, Without Extra Training Data30

18.70, With Extra Training Data

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.11

Chapter 2 Preview

13Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageS E M A N T I C S E G M E N TAT I O N

A DEMONSTRATION OF SEMANTIC SEGMENTATION
Source: Visual Object Classes Challenge, 2012

Semantic segmentation is the task of assigningindividual image pixels a category (such as person,
bicycle, or background) (Figure 2.1.12). A plethoraof real world domains require pixel-level image
segmentation such as autonomous driving (identifying
which parts of the image a car sees are pedestrians andwhich parts are roads), image analysis (distinguishing
the foreground and background in photos), andmedical diagnosis (segmenting tumors in lungs).

Figure 2.1.12

Cityscapes
The Cityscapes dataset contains images of urban street
environments from 50 cities, taken during the daytime indifferent seasons, and it allows for evaluation on a wide
range of semantic segmentation tasks (instance-level,
panoptic, and 3D vehicle).
The task to which most researchers submit is pixel-level
semantic labeling, a challenge in which AI systems must
semantically label an image on a per-pixel level. Challengers
are evaluated on the intersection-over-union (IoU)

metric, with a higher IoU score corresponding to better
segmentation accuracy. In practical terms, a higher scoremeans that a greater proportion of the image segments
predicted by the model overlap with the image’s actual
segments.
The top-performing AI systems on Cityscapes in 2021
reported scores that are 14.6 percentage points higher thanthose in 2015. As with other computer vision tasks, the top
performing models on Cityscapes have been pretrained on
additional training data in the last few years.

CITYSCAPES CHALLENGE, PIXEL-LEVEL SEMANTIC LABELING TASK: MEAN INTERSECTION-OVER-UNION (IOU)
Source: Cityscapes Challenge, 2021 | Chart: 2022 AI Index Report90%

Mean Intersection-Over-Union (mIoU)

86.20%, With Extra Training Data

84.30%, Without Extra Training Data

80%

70%

60%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.13

Chapter 2 Preview

14Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageM E D I CA L I M AG E S E G M E N TAT I O N

A DEMONSTRATION OF KIDNEY SEGMENTATION

Medical image segmentation refers to the ability of AI
systems to segment objects of interest, such as organs,lesions, or tumors, in medical images (Figure 2.1.14).
Technical progress in this task is vital to streamlining
medical diagnoses. Advances in medical image
segmentation mean doctors can spend less time on
diagnosis and more time treating patients.

Source: Kidney and Kidney Tumor Segmentation, 2021

CVC-ClinicDB and Kvasir-SEG
CVC-ClinicDB is a dataset that includes over 600 highresolution images from 31 colonoscopies. Kvasir-SEG is a
public dataset of 1,000 high-resolution gastrointestinal polypimages that were manually segmented by doctors and crossverified by professional gastroenterologists. Both datasets
are used to track progress in medical image segmentation.
Performance is measured in mean DICE, which represents theaverage overlap between the polyp segments identified by an
AI system and the actual polyp segments.
AI systems are now capable of correctly segmenting
colonoscopy polyps at a rate of 94.2% on CVC-ClinicDB,
representing an 11.9 percentage point improvement
since 2015, and a 1.8 percentage improvement since

Figure 2.1.14

2020 (Figure 2.1.15). Similar progress has been made on
Kvasir-SEG, where presently the top-performing AI model
can accurately segment gastrointestinal polyps at a rateof 92.2%. The top results on both the CVC-ClinicDB andKvasir-SEG benchmarks were achieved by the MSRF-Net
model, one of the first convolutional neural networks
designed specifically for medical image segmentation.

CVC-CLINICDB: MEAN DICEKVASIR-SEG: MEAN DICE

Source: Papers with Code, 2021; arXiv, 2021 |Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

95%

95%

94.20%

92.17%

Mean DICE

90%

Mean DICE

90%

85%

85%

80%

80%
2015

2017

2019

2021
Figure 2.1.15a

Chapter 2 Preview

2015

2017

2019

2021
Figure 2.1.15b

15

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

The Kvasir-SEG benchmark also points to the explosion ofinterest in medical image segmentation. Prior to 2020, thedataset was referenced in only three academic papers. In
2020 that number rose to six, and in 2021 it shot up to 25.
Last year also saw the hosting of KiTS21 (the Kidney andKidney Tumor Segmentation Challenge), which challenged
medical researchers from academia and industry to create
the best systems for automatically segmenting renal
tumors and the surrounding anatomy of kidneys.

FACE DETECTION AND RECOGNITION
In facial detection, AI systems are tasked with identifying
individuals in images or videos. Although facial recognition
technology has existed for several decades, the technicalprogress in the last few years has been significant. Some of
today’s top-performing facial recognition algorithms have anear 100% success rate on challenging datasets.
Facial recognition can be used in transportation to facilitatecross-border travel, in fraud prevention to protect sensitive
documents, and in online proctoring to identify illicit
examination behavior. The greatest practical promise of
facial recognition, however, is in its potential to aid security,

which makes the technology extremely appealing tomilitaries and governments all over the world (e.g., 18 outof 24 U.S. government agencies are already using somekind of facial recognition technology).

National Institute of Standards and Technology(NIST) Face Recognition Vendor Test (FRVT)The National Institute of Standards and Technology’s
Face Recognition Vendor Test measures how well facialrecognition algorithms perform on a variety of homeland
security and law enforcement tasks, such as facerecognition across photojournalism images, identificationof child trafficking victims, deduplication of passports,
and cross-verification of visa images. Progress on facialrecognition algorithms is measured according to the false
non-match rate (FNMR) or the error rate (the frequency
with which a model fails to match an image to a person).
In 2017, some of the top-performing facial recognition
algorithms had error rates of over 50.0% on certain FRVT
tests. As of 2021, none has posted an error rate greater than
3.0%. The top-performing model across all datasets in 2021
(visa photos) registered an error rate of 0.1%, meaning thatfor every 1,000 faces, the model correctly identified 999.

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY (NIST) FACE RECOGNITION VENDOR TEST (FRVT):VERIFICATION ACCURACY by DATASET
Source: National Institute of Standards and Technology, 2021 | Chart: 2022 AI Index Report

1

False Non-Match Rate: FMNR (Log-Scale)

0.5

0.2
0.1
0.05
0.0297, WILD Photos FNMR @ FMR

0.00001

0.02
0.01
0.005

0.0044, BORDER Photos FNMR @ FMR = 0.000001
0.0023, VISABORDER Photos FNMR@FMR 0.000001
0.0022, MUGSHOT Photos FNMR @ FMR 0.00001
0.0021, MUGSHOT Photos FNMR @ FMR 0.00001 DT>=12 YRS
0.0013, VISA Photos FNMR @ FMR 0.000001

0.002
0.001
2017

2018

2019

2020

2021
Figure 2.1.16

Chapter 2 Preview

16Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

FACE DETECTION:
EFFECTS OF MASK-WEARING
Face Recognition Vendor Test (FRVT):
Face-Mask Effects
Facial recognition has become a more challenging
task with the onset of the COVID-19 pandemic andaccompanying mask mandates. The face-mask effects test
asks AI models to identify faces on two datasets of visa
border photos, one of which includes masked faces, theother which does not.
Three important trends can be gleaned from the FRVT
face-mask test: (1) Facial recognition systems still perform
relatively well on masked faces; (2) the performance onmasked faces is worse than on non-masked faces; and (3)the gap in performance has narrowed since 2019.

Although facial recognition
technology has existed
for several decades, thetechnical progress in thelast few years has beensignificant. Some of today’s
top-performing facialrecognition algorithms havea near 100% success rate on
challenging datasets.

NIST FRVT FACE MASK EFFECTS: FALSE-NON MATCH RATE
Source: National Institute of Standards and Technology, 2021 | Chart: 2022 AI Index Report

False Non-Match Rate: FMNR (Log-Scale)

0.025

0.020

0.015
0.014, Masked

0.010

0.005
0.002, Non-masked

0.000
2019

2020

2021
Figure 2.1.17

Chapter 2 Preview

17Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Masked Labeled Faces in the Wild (MLFW)
In 2021, researchers from the
Beijing University of Posts andTelecommunications released afacial recognition dataset of 6,000
masked faces in response to the newrecognition challenges posed bylarge-scale mask-wearing.

EXAMPLES OF MASKED FACES IN
THE MASKED LABELED FACES IN
THE WILD (MLFW) DATABASE
Source: Wang et al., 2021

Figure 2.1.18

As part of the dataset release, the researchers ran a series of existing state-of-the-art detection
algorithms on a variety of facial recognition datasets, including theirs, to determine how much detection
performance decreased when faces were masked. Their estimates suggest that top methods perform 5
to 16 percentage points worse on masked faces compared to unmasked ones. These findings somewhat
confirm the insights from the FRVT face-mask tests: Performance deteriorates when masks are included,
but not by an overly significant degree.
STATE-OF-THE-ART FACE DETECTION METHODS on MASKED LABELED FACES IN THE WILD (MLFW): ACCURACY
Source: Wang et. al, 2021 | Chart: 2022 AI Index Report

Face Detection Method / Dataset
ArcFace1

ArcFace3

Arcface4

CosFace2Curricularface5

SFace6

100%
91%

90%

90%

Accuracy (%)

85%

91%

83%

80%
75%

70%

60%

MLFW

SLLFW8LFW7CPLFW10CALFW11MLFW

SLLFW8LFW7CALFW11CPLFW10

MLFW

LFW7

SLLFW8CPLFW10MLFW

CALFW11SLLFW8LFW7CPLFW10CALFW11MLFW

SLLFW8LFW7CALFW11CPLFW10

MLFW

LFW7

SLLFW8CPLFW10CALFW1150%

Figure 2.1.19

Chapter 2 Preview

18Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageVISUAL REASONING
Visual reasoning assesses how well
AI systems can reason across a
combination of visual and textual
data. Visual reasoning skills are
essential in developing AI that canreason more broadly. Existing AI canalready execute certain narrow visualtasks better than humans, such asclassifying images, detecting faces,and segmenting objects. But many AI
systems struggle when challenged toreason more abstractly—for example,
generating valid inferences about theactions or motivations of agents in animage (Figure 2.1.20).

Visual Question Answering(VQA) Challenge

AN EXAMPLE OF A VISUAL REASONING TASK
Source: Goyal et al., 2021

Figure 2.1.20

SAMPLE QUESTIONS IN THE VISUAL QUESTIONANSWERING (VQA) CHALLENGE
Source: Goyal et al., 2017

In the Visual Question Answering
challenge, AI systems are tasked withanswering open-ended questions
about images (Figure 2.1.21). To
answer the questions at a high level,
AI systems must have a combined
understanding of language, vision,and commonsense reasoning.

Figure 2.1.21

Chapter 2 Preview

19Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

In the six years since the VQA challenge began, there has
been a 24.4 absolute percentage point improvement instate-of-the-art performance. In 2015, the top-performing

systems could correctly answer only 55.4% of questions(Figure 2.1.22). As of 2021, top performance stood at
79.8%—close to the human baseline of 80.8%.

VISUAL QUESTION ANSWERING (VQA) CHALLENGE: ACCURACY
Source: VQA Challenge, 2021 | Chart: 2022 AI Index Report

80.80%, Human Baseline80%

Accuracy (%)

79.78%

70%

60%

50%
2015

2016

2017

2018

2019

2020

2021
Figure 2.1.22

Chapter 2 Preview

20Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Video analysis concerns reasoning or task operation across sequential frames (videos), rather than single frames (images). Video
computer vision has a wide range of use cases, which include assisting criminal surveillance efforts, sports analytics, autonomous
driving, navigation of robots, and crowd monitoring.

2.2 COMPUTER VISION—VIDEOAC T I V I T Y R E C O G N I T I O N​A fundamental subtask in video computer vision is
activity recognition: identifying the activities that occur
in videos. AI systems have been challenged to classify
activities that range from simple actions, like walking,waving, or standing, to ones that are more complex andcontain multiple steps, like preparing a salad (which
requires an AI system to recognize and chain together
discrete actions like cutting tomatoes, washing thegreens, applying dressing, etc.)

Kinetics-400, Kinetics-600, Kinetics-700
Kinetics-400, Kinetics-600, and Kinetics-700 are a series ofdatasets for benchmarking video activity recognition. Eachdataset includes 650,000 large-scale, high-quality video
clips from YouTube that display a wide range of humanactivities, and asks AI systems to classify an action from apossible set of 400, 600, and 700 categories, respectively.
Some of the new and more challenging activity classes
added to the Kinetics-700 series include pouring wine,
playing the oboe, and making latte art.

EXAMPLE CLASSES FROM THE KINETICS DATASET
Source: Kay et al., 2017

Figure 2.2.1

Chapter 2 Preview

21Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

As of 2022, one model tops all three Kinetics datasets.
MTV, a collaboration between Google Research, Michigan
State University, and Brown University, released in January2022, achieved a 89.6% Top-1 accuracy on the 600 series,
89.1% accuracy on the 400 series, and 82.20% accuracy
on the 700 series (Figure 2.2.2). The most striking aspect
about technical progress on Kinetics is how rapidly the gap

has narrowed between performance on the datasets. In
2020, the gap between performance on Kinetics-400 andKinetics-700 was 27.14 percentage points. In one shortyear, that gap has narrowed to 7.4 points, which meansthat performance on the newer, harder dataset is occurring
more rapidly than performance on the easier dataset andsuggests that the easier ones are starting to asymptote.

KINETICS-400, KINETICS-600, KINETICS-700: TOP-1 ACCURACY
Source: Papers with Code, 2021; arXIv, 2021 | Chart: 2022 AI Index Report

89.60%, Kinetics-600
89.10%, Kinetics-400

90%82.20%, Kinetics-700

Top-1 Accuracy (%)

80%

70%

60%

50%
2016

2017

2018

2019

2020

2021

2022
Figure 2.2.2

Chapter 2 Preview

22Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

ActivityNet: Temporal Action Localization Task
ActivityNet is a video dataset for human activity
understanding that contains 700 hours of videos ofhumans doing 200 different activities (long jump, dog
walking, vacuuming, etc.). For an AI system to successfullycomplete the ActivityNet Temporal Action Localization
Task (TALT) task, it has to execute two separate steps: (1)localization (identify the precise interval during which theactivity occurs); and (2) recognition (assign the correct
category label). Temporal action localization is one of

the most complex and difficult tasks in computer vision.
Performance on TALT is measured in terms of mean average
precision, with a higher score indicating greater accuracy.
As of 2021 the top-performing model on TALT, developed
by HUST-Alibaba, scores 44.7%, a 26.9 percentage point
improvement over the top scores posted in 2016 when thechallenge began (Figure 2.2.3). Although state-of-the-art
results on the task have been posted for each subsequent
year, the gains have become increasingly small.

ACTIVITYNET, TEMPORAL ACTION LOCALIZATION TASK: MEAN AVERAGE PRECISION (mAP)
Source: ActivityNet, 2021 | Chart: 2022 AI Index Report

44.67%

Mean Average Precision (mAP)

40%

30%

20%

2016

2017

2018

2019

2020

2021
Figure 2.2.3

Chapter 2 Preview

23Artificial IntelligenceIndex Report 2022

O B J E C T D E T E CT I O N
Object detection is the task of identifying objects within
an image (Figure 2.2.4). There are different philosophies
bearing on priority, speed, and accuracy that guide thedesign of object detection systems. Systems that train
quickly might be more efficient but are less accurate.
Those that are more accurate might perform better but

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

take longer to process a video. This tradeoff between
speed and accuracy is also reflected in the types of object
detection methods pioneered in the last decade. There
are one-stage methods which prioritize speed, such as
SSD, RetinaNet, and YOLO, and two-stage methods whichprioritize accuracy, such as Mask R-CNN, Faster R-CNN,
and Cascade R-CNN.

A DEMONSTRATION OF HOW OBJECT DETECTION APPEARS TO AI SYSTEMS
Source: COCO, 2020

Figure 2.2.4

Chapter 2 Preview

24Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Common Object in Context (COCO)
Microsoft’s Common Object in Context (COCO) object
detection dataset contains over 328,000 images across morethan 80 object categories. There are many accuracy metrics
used to track performance on object detection, but for thesake of consistency, this section and the majority of thisreport considers mean average precision (mAP50).

Since 2016, there has been a 23.8 percentage point
improvement on COCO object detection, with this year’s
top model, GLIP, registering a mean average precision of
79.5%.2  Figure 2.2.5 illustrates how the use of extra training
data has taken over object detection, much as it has withother domains of computer vision.

COCO-TEST-DEV: MEAN AVERAGE PRECISION (mAP50)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

80%

79.50%, With Extra Training Data

Mean Average Precision (mAP50)

75%
77.10%, Without Extra Training Data

70%

65%

60%

55%

50%
2015

2016

2017

2018

2019

2020

2021
Figure 2.2.5

2 GLIP (Grounded Language-Image Pretraining), a model designed to master the learning of language contextual visual representations, was a collaboration of researchers from UCLA, Microsoft Research, University of Washington, University of Wisconsin–Madison, Microsoft Cloud, Microsoft AI, and International Digital Economy Academy.

Chapter 2 Preview

25Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—VideoYou Only Look Once (YOLO)You Only Look Once is an open-source object detection
model that emphasizes speed (inference latency) overabsolute accuracy.
Over the years, there have been different iterations of YOLO,
and Figure 2.2.6 plots the performance of YOLO object
detectors versus the absolute top performing detectors

on the COCO dataset. YOLO detectors have becomemuch better in terms of performance since 2017 (by 28.4
percentage points). Second, the gap in performance
between YOLO and the best-performing object detectors
has narrowed. In 2017 the gap stood at 11.7%, and it
decreased to 7.1% in 2021. In the last five years, object
detectors have been built that are both faster and better.

STATE OF THE ART (SOTA) vs. YOU ONLY LOOK ONCE (YOLO): MEAN AVERAGE PRECISION (mAP50)
Source: arXiv, 2021; GitHub, 2021 | Chart: 2022 AI Index Report

79.50%, SOTA

Mean Average Precision (mAP50)

80%

70%

72.40%, YOLO

60%

50%

40%
2016

2017

2018

2019

2020

2021
Figure 2.2.6

Chapter 2 Preview

26Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Visual Commonsense Reasoning (VCR)
The Visual Commonsense Reasoning challenge is arelatively new benchmark for visual understanding. VCRasks AI systems to answer challenging questions about
scenarios presented from images, and also to provide thereasoning behind their answers (unlike the VQA challenge,

which only requires an answer). The dataset contains
290,000 pairs of multiple-choice questions, answers,
and rationales from 110,000 image scenarios taken frommovies. Figure 2.2.7 illustrates the kinds of questions
posed in the VCR.

A SAMPLE QUESTION OF THE VISUAL COMMONSENSE REASONING (VCR) CHALLENGE
Source: Zellers et al., 2018

Figure 2.2.7

Performance on VCR is measured in the Q->AR score, which
aggregates how well machines can choose the right answer
for a given multiple-choice question (Q->A) and then selectthe correct rationale for the answer (Q->R).
Since the challenge debuted, AI systems have become
better at visual commonsense reasoning, although they stilllag far behind human levels of performance (Figure 2.2.8).
At the end of 2021, the best mark on VCR stood at 72.0, ascore that represents a 63.6% increase in performance since
2018. Although progress has been made since the challenge
was launched, improvements have become increasingly
marginal, suggesting that new techniques may need to beinvented to significantly improve performance.

VISUAL COMMONSENSE REASONING (VCR) TASK: Q->AR SCORE
Source: VCR Leaderboard, 2021 | Chart: 2022 AI Index Report90
85.00, Human Baseline

80

Q->AR Score

72.00

70

60

50

40
2018

2019

2020

2021
Figure 2.2.8

Chapter 2 Preview

27Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Natural language processing (NLP) is a subfield of AI, with roots that stretch back as far as the 1950s. NLP involves research into systems thatcan read, generate, and reason about natural language. NLP evolved from a set of systems that in its early years used handwritten rules andstatistical methodologies to one that now combines computational linguistics, rule-based modeling, statistical learning, and deep learning.
This section looks at progress in NLP across several language task domains, including: (1) English language understanding; (2) text
summarization; (3) natural language inference; (4) sentiment analysis; and (5) machine translation. In the last decade, technical progress in
NLP has been significant: The adoption of deep neural network–style machine learning methods has meant that many AI systems can now
execute complex language tasks better than many human baselines.

2.3 LANGUAGEE N G L I S H L A N G UAG E
U N D E R S TA N D I N GEnglish language understanding challenges AI systems
to understand the English language in various contexts,
such as sentence understanding, yes/no reading
comprehension, reading comprehension with logicalreasoning, etc.

SuperGLUE
SuperGLUE is a single-number metric that tracks technicalprogress on a diverse set of linguistic tasks (Figure 2.3.1).

As part of the benchmark, AI systems are tested on eight
different tasks (such as answering yes/no questions,
identifying causality in events, and doing commonsense
reading comprehension), and their performance on thesetasks is then averaged into a single score. SuperGLUE is
the successor to GLUE, an earlier benchmark that also
tests on multiple tasks. SuperGLUE was released in May
2019 after AI systems began to saturate the GLUE metric,
creating demand for a harder benchmark.

A SET OF SUPERGLUE TASKS3
Source: Wang et al., 2019

Figure 2.3.1

3  For the sake of brevity, this figure only displays 3 of the 8 tasks.

Chapter 2 Preview

28Artificial IntelligenceIndex Report 2022

At the top of the SuperGLUE leaderboard sits the SS-MoE
model with a state-of-the-art score of 91.0 (Figure 2.3.2),
which exceeds the human performance score of 89.8
given by the SuperGLUE benchmark developers. The fact

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Languagethat progress on SuperGLUE was achieved so rapidly
suggests that researchers will need to develop more
complex suites of natural language tasks to challenge
the next generation of AI systems.

SUPERGLUE: SCORE
Source: SuperGLUE Leaderboard, 2021 | Chart: 2022 AI Index Report

92

91.00

Score

91

90

89.80, Human Performance

89

88
2019

2020

2021
Figure 2.3.2

Stanford Question Answering Dataset (SQuAD)
The Stanford Question Answering Dataset (SQuAD)
benchmarks performance on reading comprehension.
The dataset includes 107,785 question-and-answer pairs
taken from 536 Wikipedia articles. Performance on SQuAD
is measured by the F1 score, which is the average overlap
between the AI system’s answers and the actual correct
answers: The higher the score, the better the performance.
As was the case with GLUE, AI systems improved so rapidly
on SQuAD that only two years after launching SQuAD in
2016, researchers released SQuAD 2.0. This second version
included more challenging reading comprehension tasks,
namely a set of 50,000 unanswerable questions that were
written in a way to appear answerable (Figure 2.3.3).

Chapter 2 Preview

HARDER QUESTIONS ADDED TO STANFORDQUESTION ANSWERING DATASET (SQUAD) 2.0
Source: Rajpurkar et al., 2018

Figure 2.3.3

29

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

At the end of 2021, the leading scores on SQuAD 1.1 and
SQuAD 2.0 stood at 95.7 and 93.2, respectively (Figure
2.3.4). Although these scores are state of the art, they
are marginal improvements over the previous year’s top

scores (0.4% and 0.2%). Both SQuAD datasets have seena trend whereby immediately after the initial launches,human-performance-exceeding scores were realized andthen followed by small, plateau-like increases.

SQUAD 1.1 and SQUAD 2.0: F1 SCORE
Source: SQuAD 1.1 and SQuAD 2.0, 2021 | Chart: 2022 AI Index Report

100

95.72, Squad 1.1

95

93.21, Squad 2.0

F1 Score

91.20, Human Baseline (v1)9089.50, Human Baseline (v2)

85

80
2016

2017

2018

2019

2020

2021
Figure 2.3.4

Reading Comprehension
Dataset Requiring Logical
Reasoning (ReClor)
The plateauing progress onbenchmarks like SQuAD suggests that
NLP models need to be tested on more
complex linguistic challenges like thoseoffered by ReClor. Created in 2020 bycomputer scientists from the National
University of Singapore, ReClor requiresthat AI systems engage in reading
comprehension that also necessitates
logical reasoning. The ReClor dataset ismade up of logical reasoning questions
from the LSAT, the entrance exam forlaw schools in the United States andCanada (Figure 2.3.5).

Chapter 2 Preview

A SAMPLE QUESTION IN READING COMPREHENSION DATASETREQUIRING LOGICAL REASONING (RECLOR)
Source: Yu et al., 2020

Figure 2.3.5

30

Artificial IntelligenceIndex Report 2022

There are two sets of questions on ReClor, easy andhard, with AI systems being judged on accuracy basedon the percentage of questions they answer correctly(Figure 2.3.6). Although AI systems are presently capableof achieving a relatively high level of performance on
the easy set of questions, they struggle on the hard set.
In 2021, the top-performing model on ReClor (hard set)scored 69.3%, roughly 22.5 percentage points lower
than the top-performing model on the easy set. Datasets
like ReClor suggest that while NLP models can executestraightforward reading comprehension tasks, they face
more difficulty when those tasks are coupled with logicalreasoning requirements.

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Although AI systems
are presently capable ofachieving a relatively highlevel of performance on theeasy set of questions, theystruggle on the hard set.

READING COMPREHENSION DATASET REQUIRING LOGICAL REASONING (RECLOR): ACCURACY
Source: ReClor Leaderboard, 2021 | Chart: 2022 AI Index Report

91.82%, Test Easy

90%

Accuracy (%)

80%

70%

69.29%, Test Hard

60%

50%
2020

2021
Figure 2.3.6

Chapter 2 Preview

31Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

T E X T S U M M A R I Z AT I O N
Text summarization is the challenge of synthesizing
a piece of text while capturing its core content.
Summarizing texts is an important component oftext classification, reading comprehension, andinformation dissemination; however, when done
manually by humans, it is time- and labor-intensive.
Developing AI systems that can functionally summarize
texts has a number of practical use cases, from aiding
universities in classifying academic papers to helping
lawyers generate case summaries.
Progress in text summarization is often scored on
ROUGE (Recall-Oriented Understudy for Gisting
Evaluation). ROUGE calculates the overlap between

a summary produced by an AI system and the reference
summary produced by a human. The higher the ROUGE
score, the greater the overlap and the more accurate thesummary.

arXiv
ArXiv is a text summarization benchmark dataset that
contains over 27,770 different papers from arXiv, theopen-access repository of scientific papers. In thefive years since benchmarking on arXiv began, AI text
summarization models have improved their performanceby 47.1% (Figure 2.3.7). However, as is the case with othernatural language benchmarks, progress seems to beplateauing.

ARXIV: ROUGE-1
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

47.15, Without Extra Training Data

ROUGE-1

45

46.74, With Extra Training Data

40

35

30
2017

2018

2019

2020

2021
Figure 2.3.7

Chapter 2 Preview

32Artificial IntelligenceIndex Report 2022

PubMedPubMed is a text summarization dataset consisting of
19,717 scientific publications from the PubMed database ofscientific papers. Progress on PubMed validates the trend
seen on arXiv: There has been significant improvement

on text classification since 2017 (34.6%), but recently thepace of that progress has slowed (Figure 2.3.8). In 2021, thetop-performing model on PubMed was HAT (hierarchicalattention transformer model), created by researchers at
Birch AI and the University of Washington.

PUBMED: ROUGE-1
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report50
48.25, With Extra Training Data

47.81, Without Extra Training Data

ROUGE-1

45

40

35
2017

2018

2019

2020

2021
Figure 2.3.8

Chapter 2 Preview

33Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

N AT U R A L L A N G UAG E I N F E R E N C E

Stanford Natural Language Inference (SNLI)Natural language inference is the task of determining
whether, given a premise, a hypothesis is true
(entailment), false (contradiction), or undetermined(neutral). This skill is also known as textual
entailment as it requires determining whether aparticular premise logically entails a hypothesis.
Natural language inference necessitates languageprocessing skills, such as named entity recognition(understanding the words you see), as well as being
able to use commonsense knowledge to distinguishbetween reasonable and unreasonable inferences.

The Stanford Natural Language Inference (SNLI) dataset
contains around 600,000 sentence pairs (premise
and associated hypothesis) that are labeled as either
entailment, contradiction, or neutral. As part of thischallenge, AI systems are asked whether premises
logically entail certain hypotheses (Figure 2.3.9).
Performance on SNLI is measured in accuracy based on
the percentage of questions answered correctly.

QUESTIONS AND LABELS IN STANFORD NATURAL LANGUAGE INFERENCE (SNLI)
Source: Bowman et al., 2015

Figure 2.3.9

Chapter 2 Preview

34Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

The top-performing model on SNLI is Facebook AI USA’s EFL, which in April 2021 posted a score of 93.1% (Figure 2.3.10).
STANFORD NATURAL LANGUAGE INFERENCE (SNLI): ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report95%

Accuracy (%)

93.10%

90%

85%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.3.10

Abductive Natural Language Inference (aNLI)
Abductive natural language inference is a moredifficult type of textual entailment. Abductive inference
requires drawing the most plausible conclusion from acontext of limited information and uncertain premises.
For instance, if Jenny were to return from work andfind her home in a disheveled state and then recall
that she left a window open, she can plausibly inferthat a burglar broke in and caused the mess.4 Althoughabduction is regarded as an essential element in how
humans communicate with one another, few studies haveattempted to study the abductive ability of AI systems.
ANLI, a new benchmark for abductive natural language
inference created in 2019 by the Allen Institute for AI,
comes with 170,000 premise and hypothesis pairs. Figure
2.3.11 illustrates the types of statements included in thedataset.

EXAMPLE QUESTIONS IN ABDUCTIVE NATURAL LANGUAGE INFERENCE (ANLI)
Source: Allen Institute for AI, 2021

Figure 2.3.11
4 This particular example of abductive commonsense reasoning is taken from Bhagavatula et al. ( 2019), the first paper that investigates the ability of AI systems to perform language-based
abductive reasoning.

Chapter 2 Preview

35Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

AI performance on abductive commonsense reasoning has increased by 7.7 percentage points since 2019; however, thetop AI systems, while close, are unable to achieve human performance levels (Figure 2.3.12). Abductive reasoning istherefore still a challenging linguistic task for AI systems.
ABDUCTIVE NATURAL LANGUAGE INFERENCE (aNLI): ACCURACY
Source: Allen Institute for AI, 2021 | Chart: 2022 AI Index Report

92.90%, Human Baseline
91.87%

92%

Accuracy (%)

90%

88%

86%

84%

2019

2020

2021
Figure 2.3.12

S E N T I M E N T A N A LYS I S
Sentiment analysis is the task of using NLP techniques
to identify the sentiment (very negative, negative,
neutral, positive, very positive) of a given text. Sentiment
analysis can be straightforward if sentences are worded
clearly and unambiguously, such as “I dislike winterweather.” However, sentiment analysis can become
more challenging when AI systems encounter sentenceswith flipped structures or negations, such as “to saythat disliking winter weather is not really my thing iscompletely inaccurate.”

Sentiment analysis has many commercial use cases, fromparsing customer reviews and field survey responses to
identifying the emotional states of customers.

SemEval 2014 Task 4 Sub Task 2
The SemEval 2014 Task 4 Sub Task 2 is a benchmark
for sentiment analysis that asks machines to engage
in sentiment analysis. This specific task tests whether
AI systems can identify the sentiment associated withparticular aspects of a text, rather than the sentiment of
entire sentences or paragraphs (Figure 2.3.13).

A SAMPLESEMEVAL TASK
Source: Pontiki et al., 2014

Figure 2.3.13

Chapter 2 Preview

36Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

The SemEval dataset is composed of 7,686 restaurant and laptop reviews, whose emotional polarities have been rated
by humans. On SemEval, AI systems are tasked with assigning the right sentiment labels to particular components ofthe text, with their performance measured in terms of the percentage of the labels they correctly assign.
In the past seven years, AI systems have become much better at sentiment analysis. As of last year, top-performing
systems estimate sentiment correctly 9 out of 10 times, whereas in 2016, they made correct estimates only 7 out of
10 times. As of 2021, the state-of-the-art scores on SemEval stood at 88.6%, realized by a team of Chinese researchers
from South China Normal University and Linklogis Co. Ltd. (Figure 2.3.14).

SEMEVAL 2014 TASK 4 SUB TASK 2: ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

90%

88.64%

Accuracy (%)

85%

80%

75%

70%
2015

2016

2017

2018

2019

2020

2021
Figure 2.3.14

M AC H I N E T R A N S L AT I O N ( M T )Machine translation examines how AI software cantranslate languages. In domains where fluency inmultiple languages is required, machine translation
can be extremely impactful. The European Union, forexample, is required to translate all its cross-national
policy documents into the 24 languages of its member
states. Using machine translators can save time, improve
efficiency, and lead to more consistent outcomes.

Chapter 2 Preview

Since 2017, neural networks have taken over machinetranslation. Unlike their predecessors, neural translators
learn from a series of prior translation tasks and predictthe likelihood of a sequence of words. Neural translation
models have revolutionized the field of machine translation
not only because they do not require human supervision,
but also because they produce the most accuratetranslations. As a result, they have been widely deployed bysearch engines and social networks.

37

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

WMT 2014, English-German and English-French
The WMT 2014 family of datasets, first introduced at the
Meeting of the Association for Computational Linguistics(ACL) 2014, consist of different kinds of translation
tasks, including translation between English-French andEnglish-German language pairs. A machine’s translation
capabilities are measured by the Bilingual Evaluation
Understudy, or BLEU, score, which compares the extent
to which a machine-translated text matches a reference
human-generated translation. The higher the score, thebetter the translation.

Both the English-French and English-German WMT 2014
benchmarks showcase the significant progress madein AI machine translation over the last decade (Figure
2.3.15). Since submissions began, there has been a 23.7%
improvement in English-French and a 68.1% improvementin English-German translation ability. Relatively speaking,although performance improvements have been moresignificant on the English-German language pair, absolutetranslation ability remains meaningfully higher on EnglishFrench translation.

WMT2014, ENGLISH-FRENCH: BLEU SCORE

WMT2014, ENGLISH-GERMAN: BLEU SCORE

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

46.40, With Extra Training Data

45

4543.95, Without Extra Training Data

40

BLEU score

BLEU Score

40

35

31.26, Without Extra Training Data

30

30

25

25

20

20
2015

2017

2019

2021

35.14, With Extra Training Data

35

2015

2017

2019

2021
Figure 2.3.15

Chapter 2 Preview

38Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Number of Commercially Available MT Systems
The growing interest in machine translation is also
reflected in the rise of commercial machine translation
services such as Google Translate. Since 2017, there
has been a nearly fivefold increase in the number ofcommercial machine translators on the market, according

to Intento (Figure 2.3.16). 2021 also saw the introduction of
three open-source machine translation services (M2M-100,mBART, and OPUS). The emergence of publicly available,
high-functioning machine translation services speaks to
the increasing accessibility of such services and bodes well
for anybody who routinely relies on translation.

NUMBER of INDEPENDENT MACHINE TRANSLATION SERVICES
Source: Intento, 2021 | Chart: 2022 AI Index Report

Number of Independent Machine Translation Services

50
46

Commercial
Open Source Pre-trained
Preview

40
34

30
38

26
23
21
28

20
16
12

10

9

18

9

10

3

3

11/2017

03/2018

21

15
3

9

8

0
05/2017

23

13

10

07/20173

07/2018

12/20183

06/2019

11/20196

07/2020

5

10/2021
Figure 2.3.16

Chapter 2 Preview

39Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.4 Speech

Another important domain of AI research is the analysis, recognition, and synthesis of human speech. In this AI subfield, AI systems are
typically rated on their ability to recognize speech and identify words and convert them into text; and also to recognize speakers andidentify the individuals speaking. Modern home assistance tools, such as Siri, are one of the many examples of commercially applied AI
speech technology.

2.4 SPEECH
English speech taken from a collection of audiobooks. On
LibriSpeech, AI systems are asked to transcribe speech
to text and then measured on word error rate, or thepercentage of words they fail to correctly transcribe.

SPEECH RECOGNITION
Speech recognition is the process of training machines
to recognize spoken words and convert them into text.
Research in this domain began at Bell Labs in the 1950s,
when the world was introduced to the automatic digit
recognition machine (named “Audrey”), which could
recognize a human saying any number from zero to nine.
Speech recognition has come a long way since then and in
the last decade has benefited tremendously from deeplearning techniques and the availability of rich speech
recognition datasets.

LibriSpeech is subdivided into two datasets. First, there
is LibriSpeech Test Clean, which includes higher-quality
recordings. Performance on Test Clean suggests how
well AI systems can transcribe speech in ideal conditions.
Second, there is LibriSpeech Test Other, which includes
lower-quality recordings. Performance on Test Other isindicative of transcription performance in environments
where sound quality is less than ideal.

Transcribe Speech: LibriSpeech (Test-Clean andOther Datasets)

AI systems perform incredibly well on LibriSpeech, somuch so that progress appears to be plateauing (Figure
2.4.1). A state-of-the-art result on the Test Clean dataset

LIBRISPEECH, TEST CLEAN: WORD ERROR RATE (WER)

LIBRISPEECH, TEST OTHER: WORD ERROR RATE (WER)

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

14

14

12

12

10

10
Word Error Rate (WER)

Word Error Rate (WER)

Introduced in 2015, LibriSpeech is a speech transcription
database that contains around 1,000 hours of 16 khz

8

6

4

3.30, Without Extra Training Data

8

6

4
1.70, Without Extra Training Data22

2.50, With Extra Training Data

1.40, With Extra Training Data

0

0
2015

2017

2019

2021

2015

2017

2019

2021
Figure 2.4.1

Chapter 2 Preview

40Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.4 Speech

was not realized in 2021, which speaks to the fact that theerror rate of the top system was already low at 1.4%. For
every 100 words that top-performing transcription models
heard, they correctly transcribed 99.
Performance on the Test Other dataset, while worse than
Test Clean, was still relatively poor. The state-of-the-art
results on Test Other were realized by the W2V-BERT
model, an MIT and Google Brain collaboration, which
posted an error rate of 2.0%.

VoxCeleb
VoxCeleb is a large-scale audiovisual dataset of humanspeech for speaker recognition, which is the task of matchingcertain speech with a particular individual. Each year, themakers of VoxCeleb host a speaker verification challenge.
A low score or equal error rate on the VoxCeleb challenge
is indicative of an AI system that makes few errors in its
attribution of speech to particular individuals.5 Figure 2.4.2
plots performance over time on VoxCeleb-1, the original
VoxCeleb dataset. Since 2017, performance on VoxCeleb has
improved: Systems that once reported equal error rates of
7.8% now report errors that are less than 1.0%.

VOXCELEB: EQUAL ERROR RATE (EER)
Source: VoxCeleb, 2021 | Chart: 2022 AI Index Report8%

Equal Error Rate (%)

6%

4%

2%

0.42%

0%
2017

2018

2019

2020

2021
Figure 2.4.2

5  The Equal Error Rate (EER) is not only a measure of the false positive rate, assigning a bad label, but also the false negative rate (failure to assign the correct label).

Chapter 2 Preview

41Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.5 Recommendation

Recommendation is the task of suggesting items that might be of interest to a user, such as movies to watch, articles to read, or products
to purchase. Recommendation systems are crucial to businesses, such as Amazon, Netflix, Spotify, and YouTube. For example, one of
the earliest open recommendation competitions in AI was the Netflix Prize; hosted in 2009, it challenged computer scientists to develop
algorithms that could accurately predict user ratings for films based on previously submitted ratings.

2.5 RECOMMENDATION
Commercial Recommendation: MovieLens 20M
The MovieLens 20M dataset contains around 20
million movie ratings for 27,000 movies from 138,000
users. The ratings are taken from MovieLens (a movie
recommendation platform), and AI systems are challenged
to see if they can predict a user’s movie preferences basedon their previously submitted ratings. The metric used to
track performance on MovieLens is Normalized Discounted
Cumulative Gain (nDCG), which is a measure of ranking

quality. A higher nDCG score means that an AI systemdelivers more accurate recommendations.
Since 2018, top models now perform roughly 5.2% betteron MovieLens 20M than they did in 2018 (Figure 2.5.1).
In 2021, the state-of-the-art system on MovieLens 20M
posted an nDCG of 0.448 and came from researchers at
the Czech Technical University in Prague.

MOVIELENS 20M: NORMALIZED DISCOUNTED CUMULATIVE GAIN@100 (nDCG@100)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Normalized Discounted Cumulative Gain@100 (nDCG@100)

0.460

0.450
0.448

0.440

0.430

0.420
2018

2019

2020

2021
Figure 2.5.1

Chapter 2 Preview

42Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.5 Recommendation

Click-Through Rate Prediction: Criteo
Click-through rate prediction is the task of predicting
the likelihood that something on a website, say anadvertisement, will be clicked. In 2014, the online
advertising platform Criteo launched an open clickthrough prediction challenge. Included as part of thischallenge dataset was information on a million ads
that were displayed during a 24-day period, whether
they were clicked, and additional information on theircharacteristics. Since the competition launched, theCriteo dataset has been widely used to test recommendersystems. On Criteo, systems are measured on area under
the curve (AUC). A higher AUC means a better clickthrough prediction rate and a stronger recommendersystem.

Performance on Criteo also indicates that recommendersystems have been slowly and steadily improving in thepast decade. Last year’s top model (Sina Weibo Corp’s
MaskNet) performed 1.8% higher on Criteo than the top
model from 2016. An improvement of 1.8% may seemsmall in absolute terms, but it can be a valuable margin in
the commercial world.
A limit of the Criteo and MovieLens benchmarks is thatthey are primarily academic measures of technicalprogress in recommendation (Figure 2.5.2). Most of theresearch work on recommendation occurs in commercial
settings. Given that companies have an incentive to keep
their recommendation improvements proprietary, theacademic metrics included in this section might not becomplete measures of the technical progress made inrecommendation.

CRITEO: AREA UNDER CURVE SCORE (AUC)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

0.813

Area Under Curve Score (AUC)

0.810

0.800

0.790
2016

2017

2018

2019

2020

2021
Figure 2.5.2

Chapter 2 Preview

43Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learning from their prioractions. Researchers train systems to optimize by rewarding them if they achieve a desired goal and then punishing them if they fail.
Systems experiment with different strategy sequences to solve their stated problem (e.g., playing chess or navigating through a maze)and select the strategies which maximize their rewards.
Reinforcement learning makes the news whenever programs like DeepMind’s AlphaZero demonstrate superhuman performance ongames like Go and Chess. However, reinforcement learning is useful in any commercial domain where computer agents need to maximize
a target goal or stand to benefit from learning from previous experiences. Reinforcement learning can help autonomous vehicles change
lanes, robots optimize manufacturing tasks, or time-series models predict future events.

2.6 REINFORCEMENT LEARNING
REINFORCEMENT LEARNINGENVIRONMENTS
A reinforcement learning environment is a computer
platform where AI agents are challenged to maximize
their performance on a defined task. Unlike other AI tasks
which require systems to train on a dataset, reinforcement
learning necessitates that AI systems have an environment
in which they can test various strategies and, in the process,identify the set of strategies that will maximize rewards.

Arcade Learning Environment: Atari-57
Introduced in 2013, the Arcade Learning Environment
is an interface that includes various Atari 2600 gameenvironments (such as “Pac-Man,” “Space Invaders,”and “Frogger”) in which AI agents are challenged to
optimize performance. To enable standard comparisons,researchers typically report average performance on the
ALE across a suite of 57 games. There are various metrics
in which performance is measured, but one of the mostcommon is mean human-normalized score. A humannormalized score of 0% represents random performance,
and a 100% score represents average human performance.
The mean human-normalized score is then the average
human-normalized score achieved by an AI system.
In late 2019, DeepMind’s MuZero algorithm achieved
state-of-the-art performance on Atari-57. MuZero
not only performed 48.3% better on Atari-57 thanthe previous best-performing model, but it also set a
new world record on Go and achieved superhumanperformance on chess and shogi.

Chapter 2 Preview

Creating reinforcement
learning models that areboth high performingand highly efficient is
an important step in thecommercial deployment ofreinforcement learning.

In 2021, however, researchers from Tsinghua University
and ByteDance launched the GDI-H3 model, whichsurpassed (and nearly doubled) MuZero’s performanceon Atari-57 (Figure 2.6.1). Moreover, GDI-H3 achieved
this performance with less training. It only used 200
million training frames, whereas MuZero used 20 billion:
GDI-H3 was twice as effective and one hundred times
more efficient. Creating reinforcement learning models
that are both high performing and highly efficient is
an important step in the commercial deployment ofreinforcement learning.

44Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

ATARI-57: MEAN HUMAN-NORMALIZED SCORE
Source: Papers with Code, 2021; arXIv, 2021 | Chart: 2022 AI Index Report

9.62

Mean-Human Normalized Score (in thousands)

10

8

6

4

2

0
2015

2016

2017

2018

2019

2020

2021
Figure 2.6.1

Procgen
Procgen is a new reinforcement learning
environment introduced by OpenAI in 2019. It
includes 16 procedurally generated video-gamelike environments specifically designed to test theability of reinforcement learning agents to learn
generalizable skills (Figure 2.6.2). Procgen wasdeveloped to overcome some of the criticisms
leveled at benchmarks like Atari that incentivized
AI systems to become narrow learners thatmaximized capacity in one particular skill.
Procgen encourages broad learning by introducing
a reinforcement learning environment thatemphasizes high diversity and forces AI systems
to train in generalizable ways. Performance on
Procgen is measured in terms of mean-normalized
score. Researchers typically train their systems on
200 million training runs and report an averagescore across the 16 Procgen games. The higher thesystem scores, the better the system.

A SCREENSHOT OF THE 16 GAME ENVIRONMENTS IN PROCGEN
Source: Cobbe et al. 2019

Figure 2.6.2

Chapter 2 Preview

45Artificial IntelligenceIndex Report 2022

In November 2021, the MuZero model from DeepMind
posted a state-of-the-art score of 0.6 on Procgen.
DeepMind’s results were a 128.6% improvement overthe baseline performance established in 2019 when

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

the environment was first released. Rapid progress onsuch a diverse benchmark signals that AI systems are
slowly improving their capacity to reason in broader
environments.

PROCGEN: MEAN-NORMALIZED SCORE
Source: arXiv, 2021 | Chart: 2022 AI Index Report

0.64

Mean-Normalized Score

0.60

0.50

0.40

0.30

2019

2020

2021
Figure 2.6.3

Chapter 2 Preview

46Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

Human Games: Chess

range of games, such as shogi and Go, and have infact beaten some of the top-ranked chess engines.
Nevertheless, looking at chess engine performance is aneffective way to relativize the progress made in AI andcompare it to a widely understandable human baseline.

Progress in reinforcement learning can also be capturedby the performance of the world’s top chess software
engines. A chess engine is a computer program that istrained to play chess at a high level by analyzing chesspositions. The performance of chess engines is ranked
on Elo, a method for identifying the relative skill levelsof players in zero-sum games like chess: A higher scoremeans a stronger player.

Computers surpassed human performance in chessa long time ago, and since then have not stopped
improving (Figure 2.6.4). By the mid-1990s, the top chessengines exceeded expert-level human performance, andby the mid-2000s they surpassed the peak performanceof Magnus Carlsen, one of the best chess players of all
time. Magnus Carlsen’s 2882 Elo, recorded in 2014, isthe highest level of human chess performance everdocumented. As of 2021, the top chess engine has
exceeded that level by 24.3%.

One caveat is that tracking the performance ofchess engines is not a complete reflection of generalreinforcement learning progress; chess engines are
specifically trained to maximize performance on chess.
Other popular reinforcement learning systems, like
DeepMind’s AlphaZero, are capable of playing a broader
CHESS SOFTWARE ENGINES: ELO SCORE
Source: Swedish Computer Chess Association, 2021 | Chart: 2022 AI Index Report

3,581

3500

3000

2882, Magnus Carlsen

Elo Score

2500

2300, Expert

2000
1700, Intermediate
1500

1000

800, Novice

500
0
1987

1992

1997

2002

2007

2012

2017

2022
Figure 2.6.4

Chapter 2 Preview

47Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

In evaluating technical progress in AI, it is relevant not only to consider improvements in technical performance but also the speed ofoperation. As this section shows, AI systems continue to improve in virtually every skill category. This performance is often realized byincreasing parameters and training systems on greater amounts of data. However, all else being equal, models that use more parametersand source more data will take longer to train. Longer train times mean slower real-world deployment. Given that the potential ofincreased training times can be offset by stronger and more robust computational infrastructures, it is important to keep track ofprogress in the hardware that powers AI systems.

2.7 HARDWARE
MLPerf: Training Time
MLPerf is an AI training competition run by the ML
Commons organization. In this challenge, participants
train systems to execute various AI tasks (image
classification, image segmentation, natural language
processing, etc.) using a common architecture. Entrants
are then ranked on their absolute wall clock time, which is
how long it takes for the system to train.
Since the MLPerf competitions began in December 2018,
two key trends have emerged: (1) Training times forvirtually every AI skill category have massively decreased;
while (2) AI hardware robustness has substantially

increased. Top-performing hardware systems can reachbaseline levels of performance in task categories likerecommendation, light-weight objection detection, image
classification, and language processing in under a minute.
Figure 2.7.2 more precisely profiles the magnitude of
improvement across each skill category since MLPerf first
introduced the category.6 For example, training times
on image classification increased roughly twenty-sevenfold, as top times dropped from 6.2 minutes in 2018 to
0.2 minutes (or 13.8 seconds) in 2021. It might be hard tofathom the magnitude of a 27-time decrease in training
time, but in real terms it is the difference between waiting
one hour for a bus versus a little more than two minutes.

MLPERF TRAINING TIME of TOP SYSTEMS by TASK: MINUTES
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

Training Time (Minutes; Log Scale)50

20
13.57, Reinforcement Learning

10
5

3.24, Object Detection (heavy-weight)
2.38, Speech recognition

2

1.26, Image Segmentation

1
0.63, Recommendation

0.5
0.34, Object Detection (light-weight)
0.23, Language Processing
0.23, Image Classi cation

0.2
2018

2019

2020

2021
Figure 2.7.1

6 The solitary point for reinforcement learning on Figure 2.7.1 indicates that a faster time was not registered in the MLPerf competitions in 2020 or 2021. The solitary points for speech recognition andimage segmentation are indicative of the fact that those AI subtask categories were added to the MLPerf competition in 2021.

Chapter 2 Preview

48Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

Top-performing hardware systems can reach baseline levelsof performance in task categories like recommendation,light-weight objection detection, image classification, andlanguage processing in under a minute.

MLPERF: SCALE of IMPROVEMENT across TASK
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

26.96

25

Scale of Improvement

22.25

20
16.47

15

10

5

0

Improvement Baseline 1.16
Reinforcement

SpeechRecognition

1.70

1.92

Language
Processing

Recommendation

2.38

Segmentation

Object Detection Object Detection(light-weight)(heavy-weight)

Image
Classi!cation
Figure 2.7.2

Chapter 2 Preview

49Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

MLPerf: Number of Accelerators
The cross-task improvements in training time are being
driven by stronger underlying hardware systems, as shownin Figure 2.7.3. Since the competition began, the highest
number of accelerators used—where an accelerator isa chip used predominantly for the machine learningcomponent of a training run, such as a GPU or a TPU—andthe mean number of accelerators used by the top systemincreased roughly 7 times while the mean number ofaccelerators used by all entrants increased 3.5 times.
Most notable, however, is the growing gap between
the average number of accelerators used by the topperforming systems and the average accelerators usedby all systems. This gap was 9 times larger at the end of2021 than it had been in 2018. This growth clearly means
that, on average, building the fastest systems requires
the most powerful hardware.

MLPERF HARDWARE: ACCELERATORS
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

4,320, Maximum Number of Accelerators Used

Number of Accelerators

4,000

3,000

1,785, Average Accelerators Used by Top System

2,000

1,000
337, Mean Number of Accelerators

012.12.201806.10.201907.29.2020

06.30.2021

12.01.2021
Figure 2.7.3

Chapter 2 Preview

50Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

IMAGENET: Training Cost

classification system. This cost is marginal, especiallywhen compared to the $1,112.6 it cost to train a similarly
performing system in 2017. In simpler terms, in four
short years, image classification training costs havedecreased by a factor of 223.

Stronger hardware has not necessarily meant costlier
training. Figure 2.7.4 plots the lowest training cost per
year on MLPerf’s image classification subtask (ImageNet).
In 2021, it cost only $4.6 to train a high-performing image

IMAGENET: TRAINING COST (to 93% ACCURACY)
Source: AI Index and Narayanan, 2021 | Chart: 2022 AI Index Report$1,000.00

Cost (in U.S. Dollars; Log Scale)$500.00$200.00$100.00$50.00$20.00$10.00
$5.00

$4.59

$2.00
$1.00
2017

2018

2019

2020

2021
Figure 2.7.4

Chapter 2 Preview

51Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.8 Robotics

In 2021, the AI Index developed a survey that asked professors who specialize in robotics at top-ranked universities around the world andin emerging economies about changes in the pricing of robotic arms as well as the uses of robotic arms in research labs. The survey wascompleted by 101 professors and researchers from over 40 universities and collected data on 117 robotic arm purchase events from 2016
to 2022. The survey results suggest that there has been a notable decline in the price of robotic arms since 2016.

2.8 ROBOTICS
Price Trends in Robotic Arms7
The survey results show a clear downward trend in thepricing of robotic arms in the last seven years. In 2017,
the median price of a robotic arm was $42,000. Since
then, the price has declined by 46.2% to reach roughly$22,600 in 2021 (Figure 2.8.1). Figure 2.8.2, which plots the
distribution of robotic arm prices, paints a similar picture:
Despite some high-priced outliers, there has been a clear
downward trend since 2017 in the price of robotic arms.

MEDIAN PRICE of ROBOTIC ARMS, 2017–21
Source: AI Index, 2022 | Chart: 2022 AI Index Report

Price (in thousands of U.S. Dollars)

$40

$30

$22.60

$20

$10

$0
2017

2018

2019

2020

2021
Figure 2.8.1

7 We have corrected Figure 2.8.1 and Figure 2.8.2 after noticing a data filtering issue with the survey result. The correct chart has since been updated. View the appendix here for links to the data. In
addition, note that academic researchers may get a discount when purchasing robotic arms so prices are lower than retail.

Chapter 2 Preview

52Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.8 Robotics

DISTRIBUTION of ROBOTIC ARM PRICES, 2017–21
Source: AI Index, 2022 | Chart: 2022 AI Index Report

Price (in thousands of U.S. Dollars)

$100

$50

$0
2017

2018

2019

2020

2021
Figure 2.8.2

AI Skills Employed by Robotics
Professors
In the survey, the AI Index also asked
robotics professors to what extent
they employ AI skills in their research.
Responses revealed that both deeplearning and reinforcement learning are
popular AI skills employed by roboticists.
More specifically, 67.0% of professors
reported using deep learning and 46.0%
reported using reinforcement learning.

AI SKILLS EMPLOYED by ROBOTICS PROFESSORS
Source: AI Index, 2022 | Chart: 2022 AI Index Report

67.00%

Deep Learning

Reinforcement
Learning

46.00%

0%

20%

40%

60%

80%

% of Respondents
Figure 2.8.3

Chapter 2 Preview

53Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

APPENDIX
ImageNet
Data on ImageNet accuracy was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (top-1 or top-5 accuracy) correspond to the result
reported in the most recent version of each paper. 	
The estimate of human-level performance is from
Russakovsky et al., 2015. Learn more about the LSVRC
ImageNet competition and the ImageNet data set.

ImageNet: Top-1 Accuracy
To highlight progress on top-1 accuracy without the use of
extra training data, scores were taken from the following
papers: 	 	
Adversarial Examples Improve Image Recognition
Billion-Scale Semi-Supervised Learning for ImageClassification
Dual Path Networks
Densely Connected Convolutional Networks
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
ImageNet Classification with Deep Convolutional Neural
Networks
Masked Autoencoders Are Scalable Vision Learners
		To highlight progress on top-1 accuracy with the use of
extra training data, scores were taken from the following
papers:Big Transfer (BiT): General Visual Representation Learning
CoAtNet: Marrying Convolution and Attention for All Data
Sizes
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Self-Training with Noisy Student Improves ImageNet
Classification
Sharpness-Aware Minimization for Efficiently Improving

Chapter 2 Preview

Generalization
Xception: Deep Learning with Depthwise Separable
Convolutions

ImageNet: Top-5 Accuracy
To highlight progress on top-5 accuracy without the use of
extra training data, scores were taken from the following
papers: 	 		Adversarial Examples Improve Image Recognition
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Exploring the Limits of Weakly Supervised Pretraining
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
GPipe: Efficient Training of Giant Neural Networks Using
Pipeline Parallelism
High-Performance Large-Scale Image Recognition Without
Normalization
ImageNet Classification with Deep Convolutional Neural
Networks
Learning Transferable Architectures for Scalable Image
Recognition
Squeeze-and-Excitation Networks
To highlight progress on top-5 accuracy with the use of
extra training data, scores were taken from the following
papers:Big Transfer (BiT): General Visual Representation Learning
Deep Residual Learning for Image Recognition
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Florence: A New Foundation Model for Computer Vision
Self-Training with Noisy Student Improves ImageNet
Classification
Xception: Deep Learning with Depthwise Separable
Convolutions

STL-10
Data on STL-10 FID scores was retrieved through adetailed arXiv literature review cross-referenced by54Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performancetechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (FID score) correspond to the result reported in themost recent version of each paper. Details on the STL-10
benchmark can be found in the STL-10 paper.

in which a method was introduced, even if it was
subsequently tested, is the year in which it is included in
the report. The reported results (accuracy) correspondto the result reported in the most recent version of each
paper. Details on the FaceForensics++ benchmark can befound in the FaceForensics++ paper.

To highlight progress on STL-10, scores were taken fromthe following papers:
DEGAS: Differentiable Efficient Generator SearchDist-GAN: An Improved GAN Using Distance Constraints
Off-Policy Reinforcement Learning for Efficient andEffective GAN Architecture
Search Score Matching Model for Unbounded Data Score

To highlight progress on FaceForensics++, scores weretaken from the following papers:
A Deep Learning Approach to Universal Image
Manipulation Detection Using a New Convolutional Layer
Detection of Deepfake Videos Using Long Distance
Attention
FakeCatcher: Detection of Synthetic Portrait Videos Using
Biological Signals
FaceForensics++: Learning to Detect Manipulated Facial
ImagesLearning Spatiotemporal Features with 3D Convolutional
Networks
Recasting Residual-Based Local Descriptors as
Convolutional Neural Networks
Rich Models for Steganalysis of Digital Images
Thinking in Frequency: Face Forgery Detection by Mining
Frequency-Aware Clues
Xception: Deep Learning with Depthwise Separable
Convolutions

CIFAR-10
Data on CIFAR-10 FID scores was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (FID score) correspond to the result reported in themost recent version of each paper. Details on the CIFAR-10
benchmark can be found in the CIFAR-10 paper.
To highlight progress on CIFAR-10, scores were taken fromthe following papers:
AutoGAN: Neural Architecture Search for Generative
Adversarial Networks
Denoising Diffusion Probabilistic Models
Improved Training of Wasserstein GANs
Large Scale GAN Training for High Fidelity Natural Image
Synthesis
Score-Based Generative Modeling in Latent Space

FaceForensics++
Data on FaceForensics++ accuracy was retrieved througha detailed arXiv literature review. The reported dates
correspond to the year during which a paper was firstpublished to arXiv or a method was introduced. With
FaceForensics, recent researchers have tested previously
existing deepfake detection methodologies. The year

Chapter 2 Preview

Celeb-DF
Data on Celeb-DF AUC was retrieved through a detailed
arXiv literature review. The reported dates correspondto the year during which a paper was first published toarXiv or a method was introduced. With Celeb-DF, recent
researchers have tested previously existing deepfake
detection methodologies. The year in which a method wasintroduced, even if it was subsequently tested, is the year
in which it is included in the report. The reported results(AUC) correspond to the result reported in the most recent
version of each paper. Details on the Celeb-DF benchmark
can be found in the Celeb-DF paper.
To highlight progress on Celeb-DF, scores were taken fromthe following papers:
Exposing DeepFake Videos by Detecting Face Warping55

Artificial IntelligenceIndex Report 2022

Artifacts
FaceForensics++: Learning to Detect Manipulated Facial
ImagesFace X-Ray for More General Face Forgery Detection
Spatial-Phase Shallow Learning: Rethinking Face Forgery
Detection in Frequency Domain

Leeds Sports Poses
Data on Leeds Sports Poses percentage of correct
keypoints (PCK) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (PCK)
correspond to the result reported in the most recent
version of each paper. Details on the Leeds Sports Poses
benchmark can be found in the Leeds Sports Poses paper.
To highlight progress on Leeds Sports Poses, scores weretaken from the following papers:
Articulated Pose Estimation by a Graphical Model with
Image Dependent Pairwise Relations
Human Pose Estimation via Convolutional Part Heatmap
Regression
Jointly Optimize Data Augmentation and Network
Training: Adversarial Data Augmentation in Human Pose
Estimation
Knowledge-Guided Deep Fractal Neural Networks for
Human Pose Estimation
OmniPose: A Multi-Scale Framework for Multi-Person Pose
Estimation
Toward Fast and Accurate Human Pose Estimation via
Soft-Gated Skip Connections

Human 3.6M
Data on Human3.6M average mean per joint position error
was retrieved through a detailed arXiv literature review
cross-referenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, and thereported results (MPJPE) correspond to the result reported
in the most recent version of each paper. Details on the

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

Human3.6M benchmark can be found in the Human3.6M
paper.
To highlight progress on Human3.6M without the use of
extra training data, scores were taken from the following
papers:3D Human Pose Estimation in Video with Temporal
Convolutions and Semi-Supervised Training
Conditional Directed Graph Convolution for 3D Human
Pose Estimation
Cross View Fusion for 3D Human Pose Estimation
Epipolar Transformers
Human3.6M: Large Scale Datasets and Predictive Methods
for 3D Human Sensing in Natural Environments
Learning 3D Human Pose from Structure and Motion
Robust Estimation of 3D Human Poses from a Single
Image
To highlight progress on Human3.6M with the use of
extra training data, scores were taken from the following
papers:Epipolar Transformers
Learnable Triangulation of Human Pose
TesseTrack: End-to-End Learnable Multi-Person
Articulated 3D Pose Tracking

Cityscapes Challenge, Pixel-Level Semantic
Labeling Task
Data on the Cityscapes challenge, pixel-level semanticlabeling task mean IoU was taken from the Cityscapes
dataset, more specifically their pixel-level semanticlabeling leaderboard. More details about the Cityscapes
dataset and other corresponding semantic segmentation
challenges can be accessed at the Cityscapes dataset
webpage.

CVC-ClinicDB and Kvasir-SEG
Data on CVC-ClinicDB and Kvasir-SEG mean dice was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code (CVC-ClinicDB and Kvasir-SEG). The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (mean56

Artificial IntelligenceIndex Report 2022dice) correspond to the result reported in the mostrecent version of each paper. Details on the CVC-ClinicDB
benchmark can be found in the CVC-ClinicDB database
page. Details on the Kvasir-SEG benchmark can be foundin the Kvasir-SEG paper.
To highlight progress on CVC-ClinicDB, scores were takenfrom the following papers:
DoubleU-Net: A Deep Convolutional Neural Network for
Medical Image Segmentation
Encoder-Decoder with Atrous Separable Convolution for
Semantic Image Segmentation
MSRF-Net: A Multi-Scale Residual Fusion Network for
Biomedical Image Segmentation
ResUNet++: An Advanced Architecture for Medical Image
Segmentation
U-Net: Convolutional Networks for Biomedical Image
Segmentation
To highlight progress on Kvasir-SEG, scores were takenfrom the following papers:
Encoder-Decoder with Atrous Separable Convolution for
Semantic Image Segmentation
MSRF-Net: A Multi-Scale Residual Fusion Network for
Biomedical Image Segmentation
PraNet: Parallel Reverse Attention Network for PolypSegmentation
U-Net: Convolutional Networks for Biomedical Image
Segmentation

National Institute of Standards and Technology(NIST) Face Recognition Vendor Test (FRVT)
and NIST FRVT Face Mask Effects
Data on NIST FRVT 1:1 verification accuracy by dataset
was obtained from the FRVT 1:1 verification leaderboard.
Data on NIST FRVT face mask effects was obtained fromthe FRVT face mask effects leaderboard. The face maskeffects leaderboard contains results of the testing of
319 face recognition algorithms that were submitted to
FRVT prior to and post mid-March 2020, when the COVID
pandemic began.

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

Visual Question Answering (VQA)
Data on VQA was taken from recent iterations of the VQA
challenge. To learn more about the VQA challenge in
general, please consult the following link. To learn more
about the 2021 iteration of the VQA challenge, please
consult the following link. More specifically, the Index
makes use of data from the following iterations of the VQA
challenge:
VQA Challenge 2016
VQA Challenge 2017
VQA Challenge 2018
VQA Challenge 2019
VQA Challenge 2020
VQA Challenge 2021

Kinetics-400, Kinetics-600, and Kinetics-700
Data on Kinetics-400, Kinetics-600, and Kinetics-700 was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers with
Code (Kinetics-400, Kinetics-600, and Kinetics-700). The
reported dates correspond to the year during which a
paper was first published to arXiv, and the reported results(accuracy) correspond to the result reported in the mostrecent version of each paper. Details on the Kinetics-400
benchmark can be found in the Kinetics-400 paper. Details
on the Kinetics-600 benchmark can be found in the
Kinetics-600 paper. Details on the Kinetics-700 benchmark
can be found in the Kinetics-700 paper.
To highlight progress on Kinetics-400, scores were takenfrom the following papers:
Co-Training Transformer with Videos and Images Improves
Action Recognition
Large-Scale Weakly-Supervised Pre-training for Video
Action Recognition
Multiview Transformers for Video Recognition
Non-Local Neural Networks
Omni-Sourced Webly-Supervised Learning for Video
Recognition
SlowFast Networks for Video Recognition
Temporal Segment Networks: Towards Good Practices for
Deep Action Recognition

57

Artificial IntelligenceIndex Report 2022

To highlight progress on Kinetics-600, scores were takenfrom the following papers:
Masked Feature Prediction for Self-Supervised Visual PreTraining
Multiview Transformers for Video Recognition
Learning Spatio-Temporal Representation with Local andGlobal Diffusion
Rethinking Spatiotemporal Feature Learning: SpeedAccuracy Trade-offs in Video Classification
SlowFast Networks for Video Recognition
To highlight progress on Kinetics-700, scores were takenfrom the following papers:
Learn to Cycle: Time-Consistent Feature Discovery for
Action Recognition
Masked Feature Prediction for Self-Supervised Visual PreTraining
Multiview Transformers for Video Recognition

ActivityNet: Temporal Action Localization Task
In the challenge, there are three separate tasks, butthey focus on the main problem of temporally localizing
where activities happen in untrimmed videos from theActivityNet benchmark. To source information on thestate-of-the-art results for TALT, the Index did a detailed
survey of arXiv papers in addition to reports of yearly
ActivityNet challenge results. More specifically, the Index
made use of the following sources of information:
TALT 2016
TALT 2017
TALT 2018
TALT 2019TALT 2020
TALT 2021

Common Object in Context (COCO)
Data on COCO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, andthe reported results (mAP50) correspond to the result
reported in the most recent version of each paper. Details

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performanceon the COCO benchmark can be found in the COCO paper.
To highlight progress on COCO without the use of extra
training data, scores were taken from the following
papers:An Analysis of Scale Invariance in Object Detection – SNIP
Deformable ConvNets v2: More Deformable, Better Results
Dynamic Head: Unifying Object Detection Heads with
Attentions
Inside-Outside Net: Detecting Objects in Context with Skip
Pooling and Recurrent Neural Networks
Mish: A Self Regularized Non-Monotonic Activation
Function
Scaled-YOLOv4: Scaling Cross Stage Partial Network
To highlight progress on COCO with the use of extra
training data, scores were taken from the following
papers:
EfficientDet: Scalable and Efficient Object Detection
Grounded Language-Image Pre-Training

You Only Look Once (YOLO)
Data on YOLO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review
and survey of GitHub repositories. The reported dates
correspond to the year during which a paper was firstpublished to arXiv or a method was introduced. Morespecifically, the Index made use of the following sourcesof information:
YOLO 2016
YOLO 2018
YOLO 2020
YOLO 2021
YOLO results for 2017 and 2019 were not included in theindex as no state-of-the-art improvements in YOLO for
those years were uncovered during the literature review
and survey of GitHub repositories.

Visual Commonsense Reasoning (VCR)
Technical progress for VCR is taken from the VCR
leaderboard; the VCR leaderboard webpage further
delineates the methodology behind the VCR challenge.
58Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

Human performance on VCR is taken from Zellers et al. (2018). Details on the VCR benchmark can be found in the VCR
paper.

SuperGLUEThe SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about the SuperGLUE benchmark
are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks and evaluation metrics for SuperGLUE are:NAME

IDENTIFIERMETRIC

Broad Coverage Diagnostics

AX-b

Matthew’s Carr

CommitmentBank

CB

Avg. F1/ Accuracy

Choice of Plausible Alternatives

COPA

Accuracy

Multi-Sentence Reading Comprehension

MultiRC

F1a/EM

Recognizing Textual Entailment

RTE

Accuracy

Words in Context

WiC

Accuracy

The Winograd Schema Challenge

WSC

Accuracy

BooIQ

BooIQ

Accuracy

Reading Comprehension with Commonsense Reasoning

ReCoRD

F1/ Accuracy

Winogender Schema Diagnostic

AX-g

Gender Parity/ Accuracy

SQuAD 1.1 and SQuAD 2.0
Data on SQuAD 1.1 performance was taken from Papers
with Code. Data on SQuAD 2.0 performance was takenfrom the SQuAD 2.0 leaderboard. Details about the SQuAD
1.1 benchmark are in the SQuAD 1.1 paper. Details about
the SQuAD 2.0 benchmark are in the SQuAD 2.0 paper.

Reading Comprehension Dataset Requiring
Logical Reasoning (ReClor)
Data on ReClor performance was taken from the ReClor
leaderboard. Details about the ReClor benchmark are in
the ReClor paper.

arXiv
Data on arXiv recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper

Chapter 2 Preview

was first published to arXiv, and the reported results(ROUGE-1) correspond to the result reported in the mostrecent version of each paper. Details about the arXiv
benchmark are in the arXiv dataset webpage.
To highlight progress on arXiv without the use of extra
training data, scores were taken from the following
papers:A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents
Extractive Summarization of Long Documents by
Combining Global and Local Context
Get to the Point: Summarization with Pointer-Generator
Networks
Systematically Exploring Redundancy Reduction in
Summarizing Long Documents
Sparsifying Transformer Models with Trainable
Representation Pooling59

Artificial IntelligenceIndex Report 2022

To highlight progress on arXiv with the use of extra
training data, scores were taken from the following
papers:Big Bird: Transformers for Longer Sequences
Hierarchical Learning for Generation with Long Source
Sequences
PEGASUS: Pre-Training with Extracted Gap-Sentences for
Abstractive Summarization

PubMedData on PubMed recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper
was first published to arXiv, and the reported results(ROUGE-1) correspond to the result reported in the mostrecent version of each paper. Details about the PubMedbenchmark are in the PubMed paper.
To highlight progress on PubMed without the use of
extra training data, scores were taken from the following
papers:A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents
Extractive Summarization of Long Documents by
Combining Global and Local Context
Get to the Point: Summarization with Pointer-Generator
Networks
Sparsifying Transformer Models with Trainable
Representation PoolingTo highlight progress on PubMed with the use of extra
training data, scores were taken from the following
papers:A Divide-and-Conquer Approach to the Summarization ofLong Documents
Hierarchical Learning for Generation with Long Source
Sequences
PEGASUS: Pre-Training with Extracted Gap-Sentences for
Abstractive Summarization

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

Stanford Natural Language Inference (SNLI)
Data on Stanford Natural Language Inference (SNLI)
accuracy was retrieved through a detailed arXiv literature
review cross-referenced by technical progress reported on
Papers with Code. The reported dates correspond to theyear during which a paper was first published to arXiv, andthe reported results (accuracy) correspond to the result
reported in the most recent version of each paper. Details
on the SNLI benchmark can be found in the SNLI paper.
To highlight progress on SNLI, scores were taken from thefollowing papers:
Compare, Compress and Propagate: Enhancing NeuralArchitectures with Alignment Factorization for Natural
Language Inference
Convolutional Neural Networks for Sentence Classification
Enhanced LSTM for Natural Language Inference
Entailment as Few-Shot Learner
Explicit Contextual Semantics for Text Comprehension
Semantics-Aware BERT for Language Understanding
Self-Explaining Structures Improve NLP Models

Abductive Natural Language Inference (aNLI)
Data on Abductive Natural Language Inference (aNLI) was
sourced from the Allen Institute for AI’s aNLI leaderboard.
Details on the aNLI benchmark can be found in the aNLI
paper.

SemEval 2014 Task 4 Sub Task 2
Data on SemEval 2014 Task 4 Sub Task 2 accuracy was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, andthe reported results (accuracy) correspond to the result
reported in the most recent version of each paper. Details
on the SemEval benchmark can be found in the SemEval
2014 paper.
To highlight progress on SemEval, scores were taken fromthe following papers:
A Multi-Task Learning Model for Chinese-Oriented Aspect
Polarity Classification and Aspect Term Extraction60Artificial IntelligenceIndex Report 2022

Aspect Level Sentiment Classification with Deep Memory
Network
Back to Reality: Leveraging Pattern-Driven Modeling to
Enable Affordable Sentiment Dependency Learning
Effective LSTMs for Target-Dependent SentimentClassification
Hierarchical Attention Based Position-Aware Network for
Aspect-Level Sentiment Analysis
Investigating Typed Syntactic Dependencies for Targeted
Sentiment Classification Using Graph Attention Neural
Network
Recurrent Attention Network on Memory for Aspect
Sentiment Analysis

WMT2014, English-French and English-German
Data on WMT2014 English-French and English-German
BLEU score was retrieved through a detailed arXiv
literature review cross-referenced by technical progress
reported on Papers with Code (English-French andEnglish-German). The reported dates correspond to theyear during which a paper was first published to arXiv, andthe reported results (BLEU score) correspond to the result
reported in the most recent version of each paper. Details
about both the WMT2014 English-French and EnglishGerman benchmarks can be found in the WMT2014 paper.
To highlight progress on WMT2014 English-French without
the use of extra training data, scores were taken from thefollowing papers:
Addressing the Rare Word Problem in Neural MachineTranslation
Google’s Neural Machine Translation System: Bridging the
Gap Between Human and Machine Translation
MUSE: Parallel Multi-Scale Attention for Sequence toSequence Learning
R-Drop: Regularized Dropout for Neural Networks
Scaling Neural Machine Translation
Understanding the Difficulty of Training Transformers
Weighted Transformer Network for Machine Translation
To highlight progress on WMT2014 English-French withthe use of extra training data, scores were taken from the

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

following papers:
Understanding Back-Translation at Scale
Very Deep Transformers for Neural Machine Translation
To highlight progress on WMT2014 English-German
without the use of extra training data, scores were takenfrom the following papers:
BERT, mBERT, or BiBERT? A Study on Contextualized
Embeddings for Neural Machine Translation
Effective Approaches to Attention-based Neural MachineTranslation
Data Diversification: A Simple Strategy for Neural MachineTranslation
Fast and Simple Mixture of Softmaxes with BPE andHybrid-LightRNN for Language Generation
Google’s Neural Machine Translation System: Bridging the
Gap Between Human and Machine Translation
Incorporating BERT into Neural Machine Translation
Weighted Transformer Network for Machine Translation
To highlight progress on WMT2014 English-German withthe use of extra training data, scores were taken from thefollowing papers:
Lessons on Parameter Sharing across Layers inTransformers
Understanding Back-Translation at Scale

Number of Commercially Available MT
Systems
Details about the number of commercially available MTsystems was sourced from Intento’s report The State ofMachine Translation, 2021. Intento is a San Franciscobased startup that analyzes commercially available MT
services.

LibriSpeech (Test-Clean and Other Dataset)
Data on LibriSpeech (Test-Clean and Other) word error
rate was retrieved through a detailed arXiv literature
review cross-referenced by technical progress reported
on Papers with Code (Test-Clean and Other). The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (word61

Artificial IntelligenceIndex Report 2022

error rate) correspond to the result reported in the mostrecent version of each paper. Details about both the
LibriSpeech Test-Clean and Test-Other benchmarks canbe found in the LibriSpeech paper.
To highlight progress on LibriSpeech Test-Clean without
the use of extra training data, scores were taken from thefollowing papers:
ASAPP-ASR: Multistream CNN and Self-Attentive SRU for
SOTA Speech Recognition
Letter-Based Speech Recognition with Gated ConvNets
Neural Network Language Modeling with Letter-BasedFeatures and Importance Sampling
SpeechStew: Simply Mix All Available Speech Recognition
Data to Train One Large Neural Network
State-of-the-Art Speech Recognition Using Multi-Stream
Self-Attention With Dilated 1D Convolutions
To highlight progress on LibriSpeech Test-Clean with theuse of extra training data, scores were taken from thefollowing papers:
Deep Speech 2: End-to-End Speech Recognition in English
and Mandarin
End-to-End ASR: From Supervised to Semi-Supervised
Learning with Modern Architectures
Pushing the Limits of Semi-Supervised Learning for
Automatic Speech Recognition
To highlight progress on LibriSpeech Test-Other without
the use of extra training data, scores were taken from thefollowing papers:
Conformer: Convolution-Augmented Transformer for
Speech Recognition
Neural Network Language Modeling with Letter-BasedFeatures and Importance Sampling
SpeechStew: Simply Mix All Available Speech Recognition
Data to Train One Large Neural Network
Transformer-Based Acoustic Modeling for Hybrid Speech
Recognition

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

To highlight progress on LibriSpeech Test-Other with theuse of extra training data, scores were taken from thefollowing papers:
Deep Speech 2: End-to-End Speech Recognition in English
and Mandarin
End-to-End ASR: From Supervised to Semi-Supervised
Learning with Modern Architectures
Pushing the Limits of Semi-Supervised Learning for
Automatic Speech Recognition
W2v-BERT: Combining Contrastive Learning and Masked
Language Modeling for Self-Supervised Speech PreTraining

VoxCeleb
VoxCeleb is an audio-visual dataset consisting of shortclips of human speech, extracted from interview videos
uploaded to YouTube. VoxCeleb contains speech from
7,000-plus speakers spanning a wide range of ethnicities,
accents, professions, and ages—amounting to over a
million utterances (face-tracks are captured “in the wild,”with background chatter, laughter, overlapping speech,
pose variation, and different lighting conditions) recorded
over a period of 2,000 hours (both audio and video). Eachsegment is at least three seconds long. The data contains
an audio dataset based on celebrity voices, shorts,
films, and conversational pieces (e.g., talk shows). The
initial VoxCeleb 1 (100,000 utterances taken from 1,251
celebrities on YouTube) was expanded to VoxCeleb 2 (1million utterances from 6,112 celebrities).
For the sake of consistency, the AI Index reported scoreson the initial VoxCeleb dataset. Specifically, the Index
made use of the following sources of information:
The IDLAB VoxSRC-20 Submission: Large Margin FineTuning and Quality-Aware Score Calibration in DNN BasedSpeaker Verification
The SpeakIn System for VoxCeleb Speaker Recognition
Challenge 2021
VoxCeleb: A Large-Scale Speaker Identification Dataset
VoxCeleb2: Deep Speaker Recognition
VoxCeleb: Large-Scale Speaker Verification in the Wild

62

Artificial IntelligenceIndex Report 2022

MovieLens 20M
Data on MovieLens 20M normalized discounted
cumulative gain@100 (nDCG@100) was retrieved througha detailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (nDCG@100) correspond to the result reported
in the most recent version of each paper. Details on theMovieLens series of benchmarks can be found in Harper et
al. 2015.
To highlight progress on MovieLens 20M, scores weretaken from the following papers:
Deep Variational Autoencoder with Shallow Parallel Path
for Top-N Recommendation (VASP)
Enhancing VAEs for Collaborative Filtering: Flexible Priors& Gating Mechanisms
RaCT: Toward Amortized Ranking-Critical Training for
Collaborative Filtering
Variational Autoencoders for Collaborative Filtering

CriteoData on Criteo area under curve score (AUC) was retrieved
through a detailed arXiv literature review cross-referenced
by technical progress reported on Papers with Code.
The reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (AUC) correspond to the result reported in themost recent version of each paper. Details on the Criteo
benchmark can be found on the Criteo Kaggle Challenge
page.
To highlight progress on Criteo, scores were taken fromthe following papers:
AutoInt: Automatic Feature Interaction Learning via SelfAttentive Neural Networks
DeepFM: A Factorization-Machine Based Neural Network
for CTR Prediction
DeepLight: Deep Lightweight Feature Interactions for
Accelerating CTR Predictions in Ad Serving
FAT-DeepFFM: Field Attentive Deep Field-aware
Factorization Machine
Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

MaskNet: Introducing Feature-Wise Multiplication to CTR
Ranking Models by Instance-Guided Mask
Product-Based Neural Networks for User Response
Prediction

Arcade Learning Environment: Atari-57
Data on Arcade Learning Environment: Atari-57 meanhuman-normalized score was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (meanhuman-normalized score) correspond to the result
reported in the most recent version of each paper. Details
on the Arcade Learning Environment: Atari-57 benchmark
can be found in the Arcade Learning Environment paper.
To highlight progress on Arcade Learning Environment:
Atari-57, scores were taken from the following papers:
Dueling Network Architectures for Deep Reinforcement
Learning
Distributional Reinforcement Learning with Quantile
Regression
GDI: Rethinking What Makes Reinforcement Learning
Different From Supervised Learning
Mastering Atari, Go, Chess and Shogi by Planning with a
Learned Model
Recurrent Experience Replay in Distributed Reinforcement
Learning

Procgen
Data on Procgen mean-normalized score was retrieved
through a detailed arXiv literature review. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (meannormalized score) correspond to the result reported in themost recent version of each paper. Details on the Procgenbenchmark can be found in the Procgen paper.
To highlight progress on Procgen, scores were taken fromthe following papers:
Automatic Data Augmentation for Generalization in
Reinforcement Learning
63

Artificial IntelligenceIndex Report 2022

Leveraging Procedural Generation to Benchmark
Reinforcement Learning
Procedural Generalization by Planning with SelfSupervised World Models

Chess
Data on the performance of chess software engines was
taken from the Swedish Chess Computer Association’s
ranking of top chess software engines. The Swedish Chess
Computer Association tests computer chess softwaresystems against one another and releases a ranking list of
the top-performing systems. The ranking list produced by
the Swedish Chess Computer Association is a statisticallysignificant and meaningful measurement of chess engine
performance because engines are pitted against one
another in thousands of tournament-like games and each
employ the same underlying hardware. Data on Magnus
Carlsen’s top ELO score was taken from the International
Chess Federation.

Training Time and Number of Accelerators
Data on training time and number of accelerators for AI
systems was taken from the MLPerf Training benchmark
competitions. More specifically, the AI Index made use ofdata from the following MLPerf training competitions:
MLPerf Training v0.5, 2018
MLPerf Training v0.6, 2019
MLPerf Training v0.7, 2020
MLPerf Training v1.0, 2021
MLPerf Training v1.1, 2021

Chapter 2 Preview

APPENDIX
Chapter 2: Technical Performance

Details on the MLPerf Training benchmark can be foundin the MLPerf Training benchmark paper. Details on
the current benchmark categories as well as technicalinformation about submission and competition
subdivisions can be found on the MLPerf Training
webpage.

ImageNet Training Cost
Data on ImageNet Training cost was based on researchfrom DAWNBench and the individual research of Deepak
Narayanan. DAWNBench is a benchmark suite for end-toend deep-learning training and inference. DAWNBenchprovides a reference set of common deep-learning
workloads for quantifying training time, training cost,inference latency, and inference cost across different
optimization strategies, model architectures, software
frameworks, clouds, and hardware. More details available
at DAWNBench.
Because DAWNBench was deprecated after March 2020,
data on the training cost for the most recent round of
MLPerf submissions was manually collected by Deepak
Narayanan.

64Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

AI Index Robotics Survey
The survey was distributed to 509 professors who specialize in robotics from 67 universities online over three waves from
December 2021 to February 2022. The selection of universities was based on the World University Rankings 2021 withgeographic representation across the globe. Please view the complete survey raw data in our public data folder here.
The survey was completed by 101 professors from 43 universities, including:​Aalborg University, Denmark
Ain Shams University, Egypt
Carnegie Mellon University, United States
Columbia University, United States
Cornell University, United States
Delft University of Technology, Netherlands
ETH Zurich, Switzerland
Hong Kong University of Science and Technology, Hong
Kong
Korea Advanced Institute of Science and Technology,
South Korea
KU Leuven, Belgium
Massachusetts Institute of Technology, United States
Nanyang Technological University, Singapore
National Polytechnic Institute, Mexico
National University of Singapore, Singapore
Peking University, China
Politecnico di Milano, Italy
Pontificia Universidad Católica de Chile, Chile
Princeton University, United States
Purdue University, United States
RWTH Aachen University, Germany
Seoul National University, South Korea

Chapter 2 Preview

Stanford University, United States
Stellenbosch University, South Africa
Swiss Federal Institute of Technology Lausanne,
Switzerland
Tokyo Institute of Technology, Japan
University of British Columbia, Canada
University of California Berkeley, United States
University of California Los Angeles, United States
University of California San Diego, United States
University of Cambridge, United Kingdom
University of Cape Town, South Africa
University College London, United Kingdom
University of Hong Kong, Hong kong
University of Illinois at Urbana-Champaign, United States
University of Malaya, Malaysia
University of Manchester, United Kingdom
University of Michigan, United States
Universitat Politècnica de Catalunya, Spain
University of Texas at Austin, United States
University of Tokyo, Japan
University of Toronto, Canada
University of Waterloo, Canada
Zhejiang University, China

65