Artificial IntelligenceIndex Report 2024

CHAPTER 2:

Technical
Performance

CHAPTER 2:

Artificial IntelligenceIndex Report 2024

Technical
Performance

Preview
Overview		

4

Editing		

28

Chapter Highlights

5

EditVal		28

Highlighted Research: ControlNet

29

Highlighted Research: Instruct-NeRF2NeRF

31

2.1 Overview of AI in 2023

6

Timeline: Significant Model Releases

6State of AI Performance

9

AI Index Benchmarks

10

2.2 Language

13

Understanding

14

HELM: Holistic Evaluation of
Language Models

14

MMLU: Massive Multitask
Language Understanding

15

Generation		
Chatbot Arena Leaderboard
Factuality and Truthfulness

16
16
18

TruthfulQA

18

HaluEval		

20

2.3 Coding		

22

Generation		

22

HumanEval

22

SWE-Bench23

2.4 Image Computer Vision andImage Generation

24

Generation		

24

HEIM: Holistic Evaluation of
Text-to-Image Models

25

Highlighted Research: MVDream

26

Instruction-Following
VisIT-Bench27
27

Segmentation
Highlighted Research: Segment Anything
3D Reconstruction From Images

33
33
35

Highlighted Research: Skoltech3D

35

Highlighted Research: RealFusion

36

2.5 Video Computer Vision andVideo Generation

37

Generation		

37

UCF101		

37

Highlighted Research: Align Your Latents

38

Highlighted Research: Emu Video39

2.6 Reasoning

40

General Reasoning

40

MMMU: A Massive Multi-discipline
Multimodal Understanding and Reasoning
Benchmark for Expert AGI

40

GPQA: A Graduate-Level Google-Proof
Q&A Benchmark

43

Highlighted Research: Comparing Humans,
GPT-4, and GPT-4V on Abstraction andReasoning Tasks
44
Mathematical Reasoning

45

GSM8K		

45

MATH		

47

PlanBench48

Visual Reasoning

49

Visual Commonsense Reasoning (VCR)

49

2

CHAPTER 2:

Artificial IntelligenceIndex Report 2024

Technical
Performance

Preview (cont’d)Moral Reasoning

50

MoCa		

50

Causal Reasoning

52

BigToM		Highlighted Research:
Tübingen Cause-Effect Pairs

2.11 Properties of LLMs

69

Highlighted Research:
Challenging the Notion of Emergent Behavior

69

52

Highlighted Research:
Changes in LLM Performance Over Time

71

54

Highlighted Research:
LLMs Are Poor Self-Correctors

73

2.7 Audio		

55

Closed vs. Open Model Performance

74

Generation		

55

2.12 Techniques for LLM Improvement76

Prompting		

76

Highlighted Research: UniAudio

56

Highlighted Research:
MusicGEN and MusicLM

57

Highlighted Research:
Graph of Thoughts Prompting

76

2.8 Agents		

59

General Agents

59

Highlighted Research:
Optimization by PROmpting (OPRO)78

AgentBench

59

Highlighted Research: Voyageur

61

Task-Specific Agents

62

MLAgentBench

62

2.9 Robotics

63

Highlighted Research: PaLM-E

63

Highlighted Research: RT-2

65

2.10 Reinforcement Learning

66

Reinforcement Learning from Human Feedback

66

Highlighted Research: RLAIF

67

Highlighted Research:
Direct Preference Optimization

68

Fine-Tuning		
Highlighted Research: QLoRA
Attention		
Highlighted Research: Flash-Decoding

79
79
80
80

2.13 Environmental Impact of AI Systems 82
General Environmental Impact

82

Training		

82

Inference		

84

Positive Use Cases

85

Appendix		

86

ACCESS THE PUBLIC DATA

3

Artificial IntelligenceIndex Report 2024

CHAPTER 2:

Technical
Performance

Overview
The technical performance section of this year’s AI Index offers a comprehensive overview
of AI advancements in 2023. It starts with a high-level overview of AI technical performance,
tracing its broad evolution over time. The chapter then examines the current state of a wide
range of AI capabilities, including language processing, coding, computer vision (image andvideo analysis), reasoning, audio processing, autonomous agents, robotics, and reinforcement
learning. It also shines a spotlight on notable AI research breakthroughs from the past year,
exploring methods for improving LLMs through prompting, optimization, and fine-tuning, andwraps up with an exploration of AI systems’ environmental footprint.

4Artificial IntelligenceIndex Report 2024

CHAPTER 2:

Technical
Performance

Chapter Highlights
1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several
benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on
more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.

2. Here comes multimodal AI. Traditionally AI systems have been limited in scope, with language models
excelling in text comprehension but faltering in image processing, and vice versa. However, recent advancements have
led to the development of strong multimodal models, such as Google’s Gemini and OpenAI’s GPT-4. These models
demonstrate flexibility and are capable of handling images and text and, in some instances, can even process audio.

3. Harder benchmarks emerge. AI models have reached performance saturation on established benchmarks
such as ImageNet, SQuAD, and SuperGLUE, prompting researchers to develop more challenging ones. In 2023, several
challenging new benchmarks emerged, including SWE-bench for coding, HEIM for image generation, MMMU for general
reasoning, MoCa for moral reasoning, AgentBench for agent-based behavior, and HaluEval for hallucinations.

4. Better AI means better data which means … even better AI. New AI models such as SegmentAnything
and Skoltech are being used to generate specialized data for tasks like image segmentation and 3D reconstruction. Data isvital for AI technical improvements. The use of AI to create more data enhances current capabilities and paves the way for
future algorithmic improvements, especially on harder tasks.

5. Human evaluation is in. With generative models producing high-quality text, images, and more, benchmarking
has slowly started shifting toward incorporating human evaluations like the Chatbot Arena Leaderboard rather thancomputerized rankings like ImageNet or SQuAD. Public feeling about AI is becoming an increasingly importantconsideration in tracking AI progress.

6. Thanks to LLMs, robots have become more flexible. The fusion of language modeling with robotics has
given rise to more flexible robotic systems like PaLM-E and RT-2. Beyond their improved robotic capabilities, these models
can ask questions, which marks a significant step toward robots that can interact more effectively with the real world.

7. More technical research in agentic AI. Creating AI agents, systems capable of autonomous operationin specific environments, has long challenged computer scientists. However, emerging research suggests that theperformance of autonomous AI agents is improving. Current agents can now master complex games like Minecraft andeffectively tackle real-world tasks, such as online shopping and research assistance.

8. Closed LLMs significantly outperform open ones. On 10 select AI benchmarks, closed models
outperformed open ones, with a median performance advantage of 24.2%. Differences in the performance of closed andopen models carry important implications for AI policy debates.

5

Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

The technical performance chapter begins with a high-level overview of significant model releases in 2023 and reviews
the current state of AI technical performance.

2.1 Overview of AI in 2023
Timeline: Significant Model Releases
As chosen by the AI Index Steering Committee, here are some of the most notable model releases of 2023.
Date

Model

Type

Creator(s)SignificanceMar. 14, 2023

ClaudeLarge language
model

Anthropic

Claude is the firstpublicly released LLM
from Anthropic, one of
OpenAI’s main rivals.
Claude is designed to be
as helpful, honest, andharmless as possible.

Mar. 14, 2023

Mar. 23, 2023

GPT-4

StableDiffusion v2

Large language
model

Text-to-image
model

OpenAI

Stability AI

GPT-4, improving
over GPT-3, is among
the most powerfuland capable LLMs todate and surpasses
human performance on
numerous benchmarks.

Image

Figure 2.1.1
Source: Anthropic, 2023

Figure 2.1.2
Source: Medium, 2023

Stable Diffusion v2 is an
upgrade of Stability AI’s
existing text-to-image
model and produceshigher-resolution,
superior-quality images.

Figure 2.1.3
Source: Stability AI, 2023

Apr. 5, 2023

Segment
Anything

Image
segmentation

Meta

Segment Anything isan AI model capableof isolating objects inimages using zero-shot
generalization.
Figure 2.1.4
Source: Meta, 2023

Chapter 2 Preview

6Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

Date

Model

Type

Creator(s)SignificanceJul. 18, 2023

Llama 2Large language
model

Meta

Llama 2, an updated
version of Meta’s flagship
LLM, is open-source. Its
smaller variants (7B and
13B) deliver relatively
high performance for
their size.

Aug. 20, 2023

DALL-E 3

Image generation

OpenAI

Image

Figure 2.1.5
Source: Meta, 2023

DALL-E 3 is an improved
version of OpenAI’s
existing text-to-visionmodel DALL-E.
Figure 2.1.6
Source: OpenAI, 2023

Aug. 29, 2023

Sep. 27, 2023

Oct. 27, 2023

Nov. 6, 2023

Chapter 2 Preview

SynthID

Mistral 7B

Ernie 4.0

GPT-4 Turbo

Watermarking

Large language
model

Large language
model

Large language
model

Google,DeepMind

Mistral AI

Baidu

OpenAI

SynthID is a tool forwatermarking AIgenerated music andimages. Its watermarks
remain detectable evenafter image alterations.

Mistral 7B, launched
by French AI company
Mistral, is a compact 7
billion parameter modelthat surpasses Llama
2 13B in performance,
ranking it top in its classfor size.
Baidu, a multinational
Chinese technology
company, has launched
Ernie 4.0, which isamong the highestperforming Chinese
LLMs to date.
GPT-4 Turbo is an
upgraded large
language model
boasting a 128K context
window and reduced
pricing.

Figure 2.1.7
Source: DeepMind, 2023

Figure 2.1.8
Source: Mistral AI, 2023

Figure 2.1.9
Source: PR Newswire, 2023

Figure 2.1.10
Source: Tech.co, 2023

7

Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

Date

Model

Type

Creator(s)SignificanceNov. 6, 2023

Whisper v3

Speech-to-text

OpenAI

Whisper v3 is an opensource speech-to-text
model known for its
increased accuracy
and extended language
support.

Nov. 21, 2023

Nov. 22, 2023

Dec. 6, 2023

Claude 2.1

Inflection-2

Gemini

Large language
model

Large language
model

Large language

Anthropic

Inflection

Googlemodel

Dec. 21, 2023

Midjourneyv6

Text-to-image
model

Midjourney

Anthropic’s latest LLM,
Claude 2.1, features anindustry-leading 200K
context window, which
enhances its capacity
to process extensive
content such as lengthy
literary works.
Inflection-2 is thesecond LLM from the
new startup Inflection,founded by DeepMind’s
Mustafa Suleyman.
Inflection-2’s launch
underscores theintensifying competitionin the LLM arena.
Gemini emerges as aformidable competitor
to GPT-4, with one of its
variants, Gemini Ultra,
outshining GPT-4 on
numerous benchmarks.

Image

Figure 2.1.11
Source: AI Business, 2023

Figure 2.1.12
Source: Medium, 2023

Figure 2.1.13
Source: Inflection, 2023

Figure 2.1.14
Source: Medium, 2023

Midjourney’s latest
update enhances user
experience with moreintuitive prompts andsuperior image quality.

Figure 2.1.15
Source: Bootcamp, 2023

Chapter 2 Preview

8Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

State of AI Performance

Over the years, AI has surpassed human baselines ona handful of benchmarks, such as image classification

As of 2023, AI has achieved levels of performancein 2015, basic reading comprehension in 2017, visualthat surpass human capabilities across a range ofreasoning in 2020, and natural language inference intasks. Figure 2.1.16 illustrates the progress of AI

2021. As of 2023, there are still some task categories

systems relative to human baselines for nine AI
benchmarks corresponding to nine tasks (e.g., image
classification or basic-level reading comprehension).1
The AI Index team selected one benchmark towhere AI fails to exceed human ability. These tend
to be more complex cognitive tasks, such as visualcommonsense reasoning and advanced-level
mathematical problem-solving (competition-level

represent each task.

math problems).

Select AI Index technical performance benchmarks vs. human performance
Source: AI Index, 2024 | Chart: 2024 AI Index report

120%

Performance relative to the human baseline (%)

100%

Human baseline

80%

60%

40%

20%

0%

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

Image classi cation (ImageNet Top-5)

Visual reasoning (VQA)

Visual commonsense reasoning (VCR)

English language understanding (SuperGLUE)

Natural language inference (aNLI)

Basic-level reading comprehension (SQuAD 1.1)

Medium-level reading comprehension (SQuAD 2.0)

Competition-level mathematics (MATH)

Multitask language understanding (MMLU)

2022

2023

Figure 2.1.162

1 An AI benchmark is a standardized test used to evaluate the performance and capabilities of AI systems on specific tasks. For example, ImageNet is a canonical AI benchmark that featuresa large collection of labeled images, and AI systems are tasked with classifying these images accurately. Tracking progress on benchmarks has been a standard way for the AI community tomonitor the advancement of AI systems.
2 In Figure 2.1.16, the values are scaled to establish a standard metric for comparing different benchmarks. The scaling function is calibrated such that the performance of the best model foreach year is measured as a percentage of the human baseline for a given task. A value of 105% indicates, for example, that a model performs 5% better than the human baseline.

Chapter 2 Preview

9Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

AI Index Benchmarks

Due to saturation, several benchmarks featured

An emerging theme in AI technical performance,
as emphasized in last year’s report, is the observed
saturation on many benchmarks, such as ImageNet,
used to assess the proficiency of AI models.
Performance on these benchmarks has stagnatedin recent years, indicating either a plateau in AI

in the 2023 AI Index have been omitted from thisyear’s report. Figure 2.1.17 highlights a selection ofbenchmarks that were included in the 2023 edition
but not featured in this year’s report.4 It also shows
the improvement on these benchmarks since 2022.
“NA” indicates no improvement was noted.

capabilities or a shift among researchers toward more
complex research challenges.3

A selection of deprecated benchmarks from the 2023 AI Index report

Source: AI Index, 2024

BenchmarkTask category

Abductive Natural Language Inference (aNLI)

Natural language inferenceYear introduced
2019

Improvement from 2022
NA

arXiv

Text summarization

2003

NA

Cityscapes Challenge

Semantic segmentation

2016

0.23%

ImageNet

Image classi cation

2009

1.54%

Kinetics-400

Activity recognition

2017

NA

Kinetics-600

Activity recognition

2018

NA

Kinetics-700

Activity recognition

2019

NA

Kvasir-SEG

Medical image segmentation

2019

1.90%

MPII

Human pose estimation

2014

NA

PubMedText summarization

2008

NA

SST-5 Fine-Grained Classi cation

Sentiment analysis

2013

NA

STL-10

Image generation

2011

NA

SuperGLUE

English language understanding

2019

NA

Visual Question Answering Challenge (VQA)

Visual reasoning

2017

NA

VoxCeleb

Speech recognition

2017

NA

Figure 2.1.17

3 Benchmarks can also saturate or see limited improvement because the problem created is hard and the corresponding performance fails to improve. The issue of benchmark saturation
discussed in this section refers more to benchmarks where performance reaches a close-to-perfection level on which it is difficult to improve.
4 For brevity, Figure 2.1.17 highlights a selection of deprecated benchmarks. Additional benchmarks that were deprecated either because there was saturation, no new state-of-the-art score
was documented, or research focus shifted away from the benchmark include: Celeb-DF (deepfake detection), CIFAR-10 (image classification), NIST FRVT (facial recognition), and Procgen(reinforcement learning).

Chapter 2 Preview

10Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

Figure 2.1.18 illustrates the year-over-yearincreases relatively soon after they are introduced,

improvement, in percent, on a selection ofthen the improvement slows. In the last few years,

benchmarks featured in the 2023 AI Index report.

many of these benchmarks have shown little or no

Most benchmarks see significant performance

improvement.

Year-over-year improvement over time on select AI Index technical performance benchmarks

Source: AI Index, 2024 | Chart: 2024 AI Index report

25%

Year-over-year improvement (%)

20%

15%

10%

5%

4.47%, Kinetics-400
3.02%, COCO
1.84%, VQA
1.54%, ImageNet Top 1
0.23%, Cityscapes
0.11%, SuperGLUE

0%
2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

2023
Figure 2.1.18

Chapter 2 Preview

11Chapter 2: Technical Performance
2.1 Overview of AI in 2023

Artificial IntelligenceIndex Report 2024

In response to benchmark saturation, AI researchers

including those for tasks in coding, advanced

are pivoting away from traditional benchmarks andreasoning, and agentic behavior—areas that were

testing AI on more difficult challenges. The 2024 AI

underrepresented in previous versions of the report

Index tracks progress on several new benchmarks(Figure 2.1.19).5

New benchmarks featured in the 2024 AI Index report
Source: AI Index, 2024

BenchmarkTask category

Year introduced

AgentBenchAgent-based behavior

2023

BigToM

Causal reasoning2023

Chatbot Arena Leaderboard

General language

2023

EditValImage editing

2023

GPQA

General reasoning

2023

GSM8K

Mathematical reasoning

2021

HEIM

Image generation

2023

HELM

General language

2021

HaluEval

Factuality

2023

HumanEval

Coding2021

MATH

Mathematical reasoning

2021

MLAgentBenchAgent-based behavior

2023

MMMU

General reasoning

2023

MoCa

Moral reasoning

2023

PlanBench

Planning

2023

SWE-bench

Coding2023

TruthfulQA

Factuality

2021

VisIT-Bench

Image instruction-following

2023
Figure 2.1.19

5 This report includes an Appendix with details regarding the sourcing of new benchmarks featured in this chapter.

Chapter 2 Preview

12Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

2.2 Language

A sample output from GPT-4
Source: AI Index, 2024

Natural language processing (NLP) enables
computers to understand, interpret,
generate, and transform text. Current stateof-the-art models, such as OpenAI’s GPT-4
and Google’s Gemini, are able to generate
fluent and coherent prose and display highlevels of language understanding ability(Figure 2.2.1). Many of these models can also
now handle different input forms, such asimages and audio (Figure 2.2.2).

Figure 2.2.1

Gemini handling image and audio inputs
Source: Google, 2024

Figure 2.2.2

Chapter 2 Preview

13Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

Understanding

In 2022, Stanford researchers introduced HELM

English language understanding challenges AI systems

to evaluate LLMs across diverse scenarios,

to understand the English language in various ways

including reading comprehension, language

such as reading comprehension and logical reasoning.

understanding, and mathematical reasoning.6(Holistic Evaluation of Language Models), designed

HELM: Holistic Evaluation of Language Models
As illustrated above, in recent years LLMs havesurpassed human performance on traditional Englishlanguage benchmarks, such as SQuAD (question
answering) and SuperGLUE (language understanding).
This rapid advancement has led to the need for morecomprehensive benchmarks.

0.78

0.77

0.73

0.72

0.69

0.68

0.60
0.40

January 2024, GPT-4 leads the aggregate HELM
leaderboard with a mean win rate of 0.96 (Figure
2.2.3); however, different models top different task

Task

Leading model

Score

GSM8K - EM

GPT-4 (0613)

0.93

LegalBench - EM

GPT-4 (0613)

0.71

MATH - Equivalent (CoT)GPT-4 Turbo (1106 preview)

0.86

MMLU - EM

GPT-4 (0613)

0.74

MedQA - EM

GPT-4 Turbo (1106 preview)

0.82

WMT 2014 - BLEU-4

Palmyra X V3 (72B)

0.26

Anthropic Claude 2.0

0.96

PaLM-2 (Bison)

0.81

GPT-4 (0613)

Anthropic Claude v1.3

PaLM-2 (Bison)

OpenbookQA - EM

Mixtral (8×7B 32K seqlen)

NaturalQuestions (open-book) - F1

Yi (34B)

0.00

PaLM-2 (Unicorn)

0.46

Palmyra X V2 (33B)

0.78

Llama 2 (70B)

Palmyra X V3 (72B)

Yi (34B)

NaturalQuestions (closed-book) F1

GPT-4 Turbo (1106 preview)

NarrativeQA - F1

0.20

GPT-4 (0613)

Mean win rate

0.80

average performance across all scenarios. As of

Source: CRFM, 2023 | Table: 2024 AI Index report

1.00 0.96
0.78

OpenAI, and uses a “mean win rate” to track

Leaders on individual HELM sub-benchmarks

Source: CRFM, 2023 | Chart: 2024 AI Index report

0.82

companies like Anthropic, Google, Meta, andcategories (Figure 2.2.4).7

HELM: mean win rate
0.83

HELM assesses models from several leading

Figure 2.2.4

Figure 2.2.3

6 HELM evaluates 10 scenarios: (1) NarrativeQA (reading comprehension), (2) Natural Questions (closed-book) (closed-book short-answer question answering), (3) Natural Questions(open-book) (open-book short-answer question answering), (4) OpenBookQA (commonsense question answering), (5) MMLU (multisubject understanding), (6) GSM8K (grade schoolmath), (7) MATH (competition math), (8) LegalBench (legal reasoning), (9) MedQA (medical knowledge), and (10) WMT 2014 (machine translation).
7 There are several versions of HELM. This section reports the score on HELM Lite, Release v1.0.0 (2023-12-19), with the data having been collected in January 2024.

Chapter 2 Preview

14Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

MMLU: Massive Multitask Language
Understanding

In early 2023, GPT-4 posted a state-of-the-art score

The Massive Multitask Language Understanding

Figure 2.2.6 highlights the top model scores on the(MMLU) benchmark assesses model performance in

MMLU benchmark in different years. The scoreszero-shot or few-shot scenarios across 57 subjects,

reported are the averages across the test set. As of

including the humanities, STEM, and social sciences

January 2024, Gemini Ultra holds the top score of(Figure 2.2.5). MMLU has emerged as a premier

90.0%, marking a 14.8 percentage point improvement

benchmark for assessing LLM capabilities: Many state-since 2022 and a 57.6 percentage point increase sinceof-the-art models like GPT-4, Claude 2, and Gemini have

MMLU’s inception in 2019. Gemini Ultra’s score wasbeen evaluated against MMLU.

the first to surpass MMLU’s human baseline of 89.8%.

on MMLU, later surpassed by Google’s Gemini Ultra.

A sample question from MMLU
Source: Hendrycks et al., 2021

Figure 2.2.5

MMLU: average accuracy

Source: Papers With Code, 2023 | Chart: 2024 AI Index report

90%

89.8%, human baseline

90.04%

Average accuracy (%)

80%

70%

60%

50%

40%

30%
2019

2020

2021

2022

2023
Figure 2.2.6

Chapter 2 Preview

15Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

Generation
In generation tasks, AI models are tested on their ability
to produce fluent and practical language responses.

the Chatbot Arena Leaderboard is one of thefirst comprehensive evaluations of public LLM
preference. The leaderboard allows users to query
two anonymous models and vote for the preferred

Chatbot Arena Leaderboard

generations (Figure 2.2.7). As of early 2024, theThe rise of capable LLMs has made it increasingly

platform has garnered over 200,000 votes, andimportant to understand which models areusers ranked OpenAI’s GPT-4 Turbo as the most

preferred by the general public. Launched in 2023,

preferred model (Figure 2.2.8).

A sample model response on the Chatbot Arena Leaderboard
Source: Chatbot Arena Leaderboard, 2024

Figure 2.2.7

Chapter 2 Preview

16Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

LMSYS Chatbot Arena for LLMs: Elo rating
Source: Hugging Face, 2024 | Chart: 2024 AI Index report

1,260
1,240

Elo rating

1,220
1,200
1,180
1,160
1,140
1,120
1,100
GP

Ge

Cla
ud
e

T-3
.5Tu
rb

o-0
613

-2.
1

mi

ni

Mi

xtr

Pro
(

De
v)

al-

8×

7b

Cla
ud
e
-In

-2.
0

str

uc
t-v
0.1

Model

Chapter 2 Preview

Cla
ud
e

-1

Mi

str

al

GP
Me

diu

m

T-4
-06
13

GP

T-4
-03
14

GP

T-4
-Tu
rb

o

Figure 2.2.8

17Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

Factuality and Truthfulness

Sample TruthfulQA questions
Source: Lin, Hilton, and Evans, 2022

Despite remarkable achievements, LLMs remain
susceptible to factual inaccuracies and content
hallucination—creating seemingly realistic, yet false,
information. The presence of real-world instanceswhere LLMs have produced hallucinations—incourt cases, for example—underscores the growing
necessity of closely monitoring trends in LLM
factuality.

TruthfulQA
Introduced at ACL 2022, TruthfulQA is a benchmark
designed to evaluate the truthfulness of LLMs ingenerating answers to questions. This benchmark
comprises approximately 800 questions across 38
categories, including health, politics, and finance.
Many questions are crafted to challenge commonly

Figure 2.2.9

held misconceptions, which typically lead humans toanswer incorrectly (Figure 2.2.9). Although one of theobservations of the paper is that larger models tend to
be less truthful, GPT-4 (RLHF) released in early 2024,
has achieved the highest performance thus far on the
TruthfulQA benchmark, with a score of 0.6 (Figure
2.2.10). This score is nearly three times higher than thatof a GPT-2-based model tested in 2021, indicating that
LLMs are becoming progressively better at providing
truthful answers.

Chapter 2 Preview

18Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

Multiple-choice task on TruthfulQA: MC1
Source: Papers with Code, 2023 | Chart: 2024 AI Index report

0.60

0.59

0.55

MC1 ↑

0.50

0.45

0.40

0.35

0.30
2021

2022

2023
Figure 2.2.10

Chapter 2 Preview

19Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

HaluEval
As previously mentioned, LLMs are prone to

A generated hallucinated QA example and ahuman-labeled ChatGPT response for a user query
Source: Li et al., 2023

hallucinations, a concerning trait given theirwidespread deployment in critical fields such as law
and medicine. While existing research has aimed tounderstand the causes of hallucinations, less effort has
been directed toward assessing the frequency of LLM
hallucinations and identifying specific content areas
where they are especially vulnerable.
HaluEval, introduced in 2023, is a new benchmark
designed to assess hallucinations in LLMs. It includes
over 35,000 samples, both hallucinated and normal,
for analysis and evaluation by LLMs (Figure 2.2.11).
The research indicates that ChatGPT fabricatesunverifiable information in approximately 19.5%
of its responses, with these fabrications spanninga variety of topics such as language, climate, andtechnology. Furthermore, the study examines how
well current LLMs can detect hallucinations. Figure
2.2.12 illustrates the performance of leading LLMs
in identifying hallucinations across various tasks,

Figure 2.2.11

including question answering, knowledge-groundeddialogue, and text summarization. The findings revealthat many LLMs struggle with these tasks, highlightingthat hallucination is a significant ongoing issue.

Chapter 2 Preview

20Chapter 2: Technical Performance
2.2 Language

Artificial IntelligenceIndex Report 2024

HaluEval hallucination classi cation accuracy
Source: Li et al., 2023 | Table: 2024 AI Index report

Models

QA

Dialogue

Summarization

GeneralChatGPT (2022)

62.59%

72.40%

58.53%

79.44%

Claude 2 (2023)

69.78%

64.73%

57.75%

75.00%

Claude (2023)

67.60%

64.83%

53.76%

73.88%

Davinci002 (2022)

60.05%

60.81%

47.77%

80.42%

Davinci003 (2022)

49.65%

68.37%

48.07%

80.40%

GPT-3 (2020)

49.21%

50.02%

51.23%

72.72%

Llama 2 (2023)

49.60%

43.99%

49.55%

20.46%

ChatGLM (2023)

47.93%

44.41%

48.57%

30.92%

Falcon (2023)

39.66%

29.08%

42.71%

18.98%

Vicuna (2023)

60.34%

46.35%

45.62%

19.48%

Alpaca (2023)

6.68%

17.55%

20.63%

9.54%
Figure 2.2.12

Chapter 2 Preview

21Chapter 2: Technical Performance
2.3 Coding

Artificial IntelligenceIndex Report 2024

Coding involves the generation of instructions that computers can follow to perform tasks. Recently, LLMs havebecome proficient coders, serving as valuable assistants to computer scientists. There is also increasing evidence thatmany coders find AI coding assistants highly useful.

2.3 Coding
Generation
On many coding tasks, AI models are challenged
to generate usable code or to solve computer

percentage point increase from the highest scorein 2022 (Figure 2.3.2). Since 2021, performance on
HumanEval has increased 64.1 percentage points.

science problems.
Sample HumanEval problem

HumanEval

Source: Chen et al., 2023

HumanEval, a benchmark for evaluating AI systems’
coding ability, was introduced by OpenAI researchers
in 2021. It consists of 164 challenging handwritten
programming problems (Figure 2.3.1). A GPT-4 model
variant (AgentCoder) currently leads in HumanEval
performance, scoring 96.3%, which is a 11.2
Figure 2.3.1

HumanEval: Pass@1

Source: Papers With Code, 2023 | Chart: 2024 AI Index report100%

96.30%

80%

Pass@1

60%

40%

20%

0%

2021

2022

2023
Figure 2.3.2

Chapter 2 Preview

22Chapter 2: Technical Performance
2.3 Coding

Artificial IntelligenceIndex Report 2024

SWE-bench
As AI systems’ coding capabilities improve, it has

multiple functions, interact with various execution

become increasingly important to benchmark

environments, and perform complex reasoning.

models on more challenging tasks. In October 2023,

Even state-of-the-art LLMs face significant

researchers introduced SWE-bench, a dataset

challenges with SWE-bench. Claude 2, thecomprising 2,294 software engineering problems

best-performing model, solved only 4.8% of thesourced from real GitHub issues and popular

dataset’s problems (Figure 2.3.4).8 In 2023, the top-

Python repositories (Figure 2.3.3). SWE-benchperforming model on SWE-bench surpassed thepresents a tougher test for AI coding proficiency,

best model from 2022 by 4.3 percentage points.

demanding that systems coordinate changes across

A sample model input from SWE-bench

SWE-bench: percent resolved

Source: Jimenez et al., 2023

Source: SWE-bench Leaderboard, 2023 | Chart: 2024 AI Index report
Unassisted

Assisted

4.80%
3.97%

4%
1.74%

Claude 2 + BM25 Retrieval

0.70%

SWE-Llama 7B + BM25 Retrieval

2022

Claude 2

SWE-Llama 13B

0%

SWE-Llama 7B

0.52%

GPT-4 + BM25 Retrieval

0.20%

ChatGPT-3.5

0%

0.70%

SWE-Llama 13B + BM25 Retrieval

2%

1.96%

GPT-4

3.01%

ChatGPT-3.5 + BM25 Retrieval

Percent resolved

6%

2023
Model
Figure 2.3.4

Figure 2.3.3

8 According to the SWE-bench leaderboard, unassisted systems have no assistance in finding the relevant files in the repository. Assisted systems operate under the “oracle” retrieval setting,which means the systems are provided with the list of files that were modified in the pull request.

Chapter 2 Preview

23Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Computer vision allows machines to understand images and videos and create realistic visuals from textual prompts
or other inputs. This technology is widely used in fields such as autonomous driving, medical imaging, and video
game development.

2.4 Image Computer Vision
and Image Generation
Generation

Which face is real?
Source: Which Face Is Real, 2023

Image generation is the task of generating
images that are indistinguishable from real ones.
Today’s image generators are so advanced thatmost people struggle to differentiate between
AI-generated images and actual images ofhuman faces (Figure 2.4.1). Figure 2.4.2 highlights
several generations from various Midjourneymodel variants from 2022 to 2024 for the
prompt “a hyper-realistic image of Harry Potter.”
The progression demonstrates the significant

Figure 2.4.1

improvement in Midjourney’s ability to generate
hyper-realistic images over a two-year period.
In 2022, the model produced cartoonish andinaccurate renderings of Harry Potter, but by 2024,
it could create startlingly realistic depictions.
Midjourney generations over time:“a hyper-realistic image of Harry Potter”
Source: Midjourney, 2023

V1, February
2022

V2, April 2022

V3, July 2022

V4, November 2022

V5, March 2023

V5.1, March 2023

V5.2, June 2023

V6, December 2023

Figure 2.4.2

Chapter 2 Preview

24Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Image-text alignment: human evaluation
Source: CRFM, 2023 | Chart: 2024 AI Index report

1.00 0.94

0.59

0.56

0.55

0.54

Openjourney v1 1B

Dreamlike Di usion v1.0 1B

Promptist + Stable Di usion v1.4 1B

MultiFusion 13B

0.40
0.20
0.00

12 key aspects crucial for real-world deployment,such as image-text alignment, image quality, andaesthetics.9 Human evaluators are used to rate themodels, a crucial feature since many automatedmetrics struggle to accurately assess various
aspects of images.

Model/adapter and number of parameters

HEIM’s findings indicate that no single model

Figure 2.4.3

excels in all criteria. For human evaluation ofimage-to-text alignment (assessing how well
the generated image matches the input text),
OpenAI’s DALL-E 2 scores highest (Figure

0.54

DeepFloyd IF X-Large 4.3B

comprehensively assess image generators across

0.60

DALL-E mega 2.6B

Image Models (HEIM), a benchmark designed to

0.63

0.60

Safe Stable Di usion strong 1B

introduced the Holistic Evaluation of Text-to-

0.72

Stable Di usion v1.4 1B

evaluation methods. In 2023, Stanford researchers

0.75

0.80

Dreamlike Photoreal v2.0 1B

prompted the development of more sophisticatedMean win rate

The rapid progress of AI text-to-image systems has

DALL-E 2 3.5B

HEIM: Holistic Evaluation of
Text-to-Image Models

Model leaders on select HEIM sub-benchmarks
Source: CRFM, 2023 | Table: 2024 AI Index report

Task

Leading model

Score

Image-text-alignment

DALL-E 2 (3.5B)

0.94

Quality

Dreamlike Photoreal v2.0 (1B)

0.92(evaluating the visual appeal), and originality

Aesthetics

Dreamlike Photoreal v2.0 (1B)

0.87

(a measure of novel image generation andOriginality

Dreamlike Photoreal v2.0 (1B)

0.98

2.4.3). In terms of image quality (gauging if theimages resemble real photographs), aesthetics

avoidance of copyright infringement), the Stable
Diffusion–based Dreamlike Photoreal model

Figure 2.4.4

ranks highest (Figure 2.4.4).

9 The 12 evaluation aspects of HEIM are: (1) Alignment: How closely does the image align with the given text? (2) Quality: What is the quality of the produced image? (3) Aesthetic:
How aesthetically pleasing is the generated image? (4) Originality: How original is the image? (5) Reasoning: Does the model understand objects, counts, and spatial relations? (6)Knowledge: Does the model have knowledge about the world? (7) Bias: Are the generated images biased? (8) Toxicity: Are the generated images toxic or inappropriate? (9) Fairness:
Do the generated images exhibit performance disparities? (10) Robust: Is the model robust to input perturbations? (11) Multilinguality: Does the model support non-English languages?
(12) Efficiency: How fast is model inference?

Chapter 2 Preview

25Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

MVDreamCreating 3D geometries or models from

Sample generations from MVDreamtext prompts has been a significant

Source: Shi et al., 2023

challenge for AI researchers, with existingmodels struggling with problems suchas multiface Janus issue (inaccurately
regenerating context described by text
prompts) and content drift (inconsistency
across different 3D views). MVDream is a
new 3D generation system developed by
ByteDance and University of California, San
Diego researchers that overcomes some ofthese hurdles (Figure 2.4.5). In quantitativeevaluations, MVDream’s generated models
achieve Inception Score (IS) and CLIP

Figure 2.4.5scores comparable to those in the training
set, indicating the high quality of thegenerated images (Figure 2.4.6). MVDream
has major implications, especially forcreative industries where 3D content
creation is traditionally time-consuming
and labor-intensive.

Quantitative evaluation on image synthesis quality
Source: Shi et al., 2023 | Table: 2024 AI Index report

Model

Batch size

FID↓

IS↑

CLIP↑

Training data

N/A

N/A

14.75 ± 0.81

31.31 ± 3.34

Multi-view Di usion - no 2D data

256

33.41

12.76 ± 0.70

30.60 ± 3.14

Multi-view Di usion - proposed

256

32.57

13.72 ± 0.91

31.40 ± 3.05

Multi-view Di usion - proposed

1024

32.06

13.68 ± 0.41

31.31 ± 3.12

Figure 2.4.6

Chapter 2 Preview

26Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Instruction-Following

VisIT-Bench
In 2023, a team of industry and academic

In computer vision, instruction-following is theresearchers introduced VisIT-Bench, a benchmark

capacity of vision-language models to interpretconsisting of 592 challenging vision-language

text-based directives related to images. For instance,

instructions across about 70 instruction categories,

an AI system could be given an image of various

such as plot analysis, art knowledge, and location

ingredients and tasked with suggesting how to use

understanding (Figure 2.4.8). As of January 2024, thethem to prepare a healthy meal. Capable instruction-

leading model on VisIT-Bench is GPT-4V, the vision-

following vision-language models are necessary forenabled variant of GPT-4 Turbo, with an Elo scoredeveloping advanced AI assistants.

of 1,349, marginally surpassing the human reference
score for VisIT-Bench (Figure 2.4.9).

A sample VisIT-Bench instruction set
Source: Bitton et al., 2023

Figure 2.4.8

VisIT-Bench: Elo rating

Source: Hugging Face, 2024 | Chart: 2024 AI Index report

1,349

1,338, human baseline
1,280
1,120

Elo rating

960
800
640
480
320
160
0

MM

GP

T

Pa
n

da

Op

GP

en

T1

3b

Mi

Fla

niG

mi

ng

o

PT
-4

Vis
ua

Oc

lG

PT

top
us

Ot
V2

ter

Ins

tru
c

Model

Chapter 2 Preview

Ly
n

tBL

IP

x(

ide

8B

)

mP

cs9

b

Lla

LU
G

ma

-O

wl

LL

Ad
a

LL

aV
A1

pte
r-v
2

3B

GP
aV
T-4
AV
Plu
s
Figure 2.4.9

27

Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Editing

A sample VisIT-Bench instruction set
Source: Bitton et al., 2023

Image editing involves using AI to modify
images based on text prompts. This AIassisted approach has broad real-world
applications in fields such as engineering,industrial design, and filmmaking.

EditValDespite the promise of text-guided image
editing, few robust methods can evaluate
how accurately AI image editors adhere to
editing prompts. EditVal, a new benchmark
for assessing text-guided image editing,includes over 13 edit types, such as adding
objects or changing their positions,
across 19 object classes (Figure 2.4.10).
The benchmark was applied to evaluate
eight leading text-guided image editing
methods including SINE and Null-text.

Figure 2.4.10

Performance improvements since 2021 on
a variety of the benchmark’s editing tasks,
are shown in Figure 2.4.11.

EditVal automatic evaluation: editing accuracy
Source: EditVal Leaderboard, 2024 | Chart: 2024 AI Index report

0.60

0.59, Object replacement
0.52, Size

0.50

Editing accuracy

0.47, Object addition
0.40
0.34, Alter parts
0.33, Average
0.30
0.25, Positional addition
0.20

0.11, Position replacement

0.10

0.00

2021

2022
Figure 2.4.11

Chapter 2 Preview

28Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Highlighted Research:

ControlNet
Conditioning inputs or performing conditional

In 2023, researchers from Stanford introduced

control refers to the process of guiding the output

a new model, ControlNet, that improves

created by an image generator by specifying

conditional control editing for large text-

certain conditions that a generated image must

to-image diffusion models (Figure 2.4.12).

meet. Existing text-to-image models often lackControlNet stands out for its ability to handle

precise control over the spatial composition

various conditioning inputs. Compared toof an image, making it difficult to use prompts

other previously released models in 2022,

alone to generate images with complex layouts,

human raters prefer ControlNet both in terms

diverse shapes, and specific poses. Fine-tuningof superior quality and better condition fidelity

these models for greater compositional control by(Figure 2.4.13). The introduction of ControlNet istraining them on additional images is theoreticallya significant step toward creating advanced text-

feasible, but many specialized datasets, such as

to-image generators capable of editing images

those for human poses, are not large enough toto more accurately replicate the complex images

support successful training.

frequently encountered in the real world.

Sample edits using ControlNet
Source: Zhang et al., 2023

Figure 2.4.12

Chapter 2 Preview

29Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

ControlNet (cont’d)
Average User Ranking (AUR): result quality and condition delity
Source: Zhang et al., 2023 | Chart: 2024 AI Index report

4.22
4
3.28

3.21

2.31
2

2022

ControlNet

ControlNet-lite

Sketch-Guided(beta = 1.6)0Sketch-Guided(beta = 3.2)

0

PITI
(sketch)

1

2023

1.02

2022

ControlNet

1.10
1ControlNet-lite

2

3

Sketch-Guided(beta = 1.6)

2.52

PITI
(sketch)Condition delity

Result quality

3

Sketch-Guided(beta = 3.2)

4

4.28

4.09

3.93

2023

Figure 2.4.13

Chapter 2 Preview

30Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Highlighted Research:

Instruct-NeRF2NeRF
New models can edit 3D geometries using(Figure 2.4.14). This method efficiently generates

only text instructions. Instruct-NeRF2NeRF is anew, edited images that adhere to textual

model developed by Berkeley researchers that

instructions, achieving greater consistency thanemploys an image-conditioned diffusion model

current leading methods (Figure 2.4.15).

for iterative text-based editing of 3D geometries
A demonstration of Instruct-NeRF2NeRF in action
Source: Haque et al., 2023

Figure 2.4.14

Chapter 2 Preview

31Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Instruct-NeRF2NeRF (cont’d)
Evaluating text-image alignment and frame consistency
Source: Haque et al., 2023 | Chart: 2024 AI Index report

1.00

1.00
0.92

0.92

0.88
0.80
CLIP direction consistency ↑CLIP text-image direction similarity ↑

0.82
0.80

0.60

0.40

0.20

0.40

0.20

0.16

0.16

0.60

0.12
0.03
0.00

SD
Sw
/ IP

On
2P

e-t

Ins

im

eD

tru
c

U

0.00

Pe
r-

t-N

eR

fra

F2

Ne

RF

me

Pe
r-

IP2

fra

P

me

On

e-t

IP2

P

im

eD

U

SD
Sw
/ IP

Ins
2P

tru
c

t-N

eR

F2

Ne

RF

Figure 2.4.15

Chapter 2 Preview

32Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Segmentation
Segmentation involves assigning individual image pixels to specific categories (for example: human, bicycle, or street).

Highlighted Research:

Segment Anything

Various segmentation masks createdby Segment Anything
Source: Kirillov et al., 2023

In 2023, Meta researchers launched Segment
Anything, a project that featured the Segment
Anything Model (SAM) and an extensive SA1B dataset for image segmentation. SAM isremarkable for being one of the first broadly
generalizable segmentation models that
performs well zero-shot on new tasks anddistributions. Segment Anything outperforms
leading segmentation methods like RITM on
16 out of 23 segmentation datasets (Figure
2.4.17). The metric on which Segment Anythingis evaluated is the mean Intersection overUnion (IoU).
Meta’s Segment Anything model was thenused, alongside human annotators, to create
the SA-1B dataset, which included over 1 billion
segmentation masks across 11 million images(Figure 2.4.16). A new segmentation dataset

Figure 2.4.16of this size will accelerate the training of
future image segmentors. Segment Anything
demonstrates how AI models can be usedalongside humans to more efficiently create
large datasets, which in turn can be used totrain even better AI systems.

Chapter 2 Preview

33

Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Segment Anything (cont’d)
SAM vs. RITM: mean IoU

DatasetSource: Kirillov et al., 2023 | Chart: 2024 AI Index report

PPDLS
BBBC038v1
DOORS
TimberSeg
NDD20
LVIS
STREETS
ZeroWaste-f
iShape
ADE20KOVISHypersim
NDISPark
VISOR
Plittersdorf
EgoHOSIBD
WoodScapeCityscapes
PIDRayDRAM
TrashCanGTEA -21.40
−25

46.90
44.70
41.10
28.90
21.10
18.50
17.30
9.10
8.80
7.80
7.00
6.10
2.70
1.80
1.50
0.80
-0.30
-0.60
-2.00
-5.80
-6.50
-15.00
−20

−15

−10

−5

0

5

10
15
Mean IoU

20

25

30

35

40

45

50

Figure 2.4.17

Chapter 2 Preview

34Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

3D Reconstruction From Images
3D image reconstruction is the process of creating three-dimensional digital geometries from two-dimensional
images. This type of reconstruction can be used in medical imaging, robotics, and virtual reality.

Highlighted Research:

Objects from the 3D reconstruction dataset

Skoltech3D

Source: Voynov et al., 2023

Data scarcity often hinders the developmentof AI systems for specific tasks. In 2023, ateam of international researchers introduced
an extensive new dataset, Skoltech3D, formultiview 3D surface reconstruction (Figure
2.4.18). Encompassing 1.4 million images of 107
scenes captured from 100 different viewpoints
under 14 distinct lighting conditions, this dataset
represents a major improvement over existing 3D
reconstruction datasets (Figure 2.4.19).

Figure 2.4.18

Skoltech3D vs. the most widely used multisensor datasets
Source: Voynov et al., 2023 | Table: 2024 AI Index report

DatasetSensor types

RGB
resolution(MPix)

Depth
resolution(MPix)

High
resolution
geometryDTU

RGB (2)

2

✓

49/64

8

80

ETH3D

RGB

24

✓

10–70

U

24

11K

TnT

RGB

8

✓

150–300

U

21

148K

BlendedMVG

unknown

3/0.4

20–1000

U

502

110K

BigBIRDRGB (5)

12

600

1

120

144K

BigBIRD

RGB-D (5)

1.2

0.3

ScanNet

RGB-D

1,3

0,3

N/A

U

1513

2.5M

Skoltech3D

RGB (2)

5

100

14

107

877K

Skoltech3D

RGB-D 1 (2)40

Skoltech3D

RGB-D 22

0.2

Skoltech3D

RGB-D 32

0.9✓

Poses/scene

Lighting# Scenes# Frames

27K

0.04

Figure 2.4.19

Chapter 2 Preview

35Chapter 2: Technical Performance
2.4 Image Computer Vision and Image Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Sample generations from RealFusion

RealFusion

Source: Melas-Kyriazi et al, 2023

RealFusion, developed by Oxford researchers,
is a new method for generating complete
3D models of objects from single images,
overcoming the challenge of often having
insufficient information from single images
for full 360 degree reconstruction. RealFusion
utilizes existing 2D image generators to produce
multiple views of an object, and then assembles
these views into a comprehensive 360 degreemodel (Figure 2.4.20). This technique yields moreaccurate 3D reconstructions compared to stateof-the-art methods from 2021 (Shelf-Supervised),across a wide range of objects (Figure 2.4.21).
Figure 2.4.20

Object reconstruction: RealFusion vs. Shelf-Supervised
Source: Melas-Kyriazi et al., 2023 | Chart: 2024 AI Index report

RealFusion (2023)

Shelf-Supervised (2021)
1.00

12.89
12.22

12

0.82
10.23

10.16

10

10.08

0.72

9.72

0.76

0.74

0.69 0.70

0.71

0.74

0.74 0.74

0.73
0.69

0.71

0.65

8.66 8.72
7.74

8 7.58
6.27

6

6.30

5.89

CLIP-similarity ↑F-score ↑

8.26

0.80

0.60

0.40

4
0.20

Vase

Teddy bear

Skateboard

OrangeMotorcycle

Chair

0.00

Backpack

Vase

Teddy bear

Skateboard

OrangeMotorcycle

Chair

0

Backpack

2

Figure 2.4.21

Chapter 2 Preview

36Chapter 2: Technical Performance
2.5 Video Computer Vision and Video Generation

Artificial IntelligenceIndex Report 2024

Video analysis concerns performing tasks across videos rather than single images.

2.5 Video Computer Vision
and Video Generation
Generation

Sample frames from UCF101
Source: Soomro et al., 2021

Video generation involves the use of AI to generate
videos from text or images.

UCF101
UCF101 is an action recognition dataset of realistic
action videos that contain 101 action categories(Figure 2.5.1). More recently, UCF101 has been used
to benchmark video generators. This year’s top
model, W.A.L.T-XL, posted an FVD16 score of 36,
more than halving the state-of-the-art score posted
the previous year (Figure 2.5.2).

UCF101: FVD16

Figure 2.5.1

Source: Papers With Code, 2023 | Chart: 2024 AI Index report

700

600

FVD16 ↓

500

400

300

200

100
36
0
2021

2022

2023
Figure 2.5.2

Chapter 2 Preview

37Chapter 2: Technical Performance
2.5 Video Computer Vision and Video Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Align Your Latents
Most existing methods can only create short, low-

GAN (LVG) in resolution quality (Figure 2.5.4).

resolution videos. To address this limitation, an

The adaptation of a text-to-image architecture tointernational team of researchers has applied latentcreate LDM, a highly effective text-to-video model,

diffusion models, traditionally used for generating

exemplifies how advanced AI techniques can behigh-quality images, to produce high-resolution

repurposed across different domains of computer

videos (Figure 2.5.3). Their Latent Diffusion Modelvision. The LDM’s strong video generation(LDM) notably outperforms previous state-of-capabilities have many real-world applications,

the-art methods released in 2022 like Long Videosuch as creating realistic driving simulations.

High-quality generation of milk dripping into a cup of coffee
Source: Blattmann et al., 2023

Figure 2.5.3

Video LDM vs. LVG: FVD and FID

Source: Blattmann et al., 2023 | Chart: 2024 AI Index report

FVD ↓
500

478

389

400
Performance

FID ↓

356

300

200

100
53.50
0

LVG
2022

31.60
Video LDM

51.90

Video LDM (cond.)
2023

Figure 2.5.4

Chapter 2 Preview

38Chapter 2: Technical Performance
2.5 Video Computer Vision and Video Generation

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Sample Emu Video generations

Emu Video

Source: Girdhar et al., 2023

Traditionally, progress in video generation has
trailed that in image generation due to its higher
complexity and the smaller datasets available
for training. Emu Video, a new transformerbased video generation model created by Meta
researchers, represents a significant step forward(Figure 2.5.5). Emu Video generates an image fromtext and then creates a video based on both thetext and image. Figure 2.5.6 illustrates the degree
to which the Emu Video model outperforms
Figure 2.5.5

previously released state-of-the-art video
generation methods. The metric is the proportion

instructions over the compared method. Emuof cases when human evaluators preferred Emu

Video simplifies the video generation process andVideo’s image quality or faithfulness to text

signals a new era of high-quality video generation.

Emu Video vs. prior works: human-evaluated video quality and text faithfulness win rate

Source: Girdhar et al., 2023 | Chart: 2024 AI Index report

Quality

100% 100%

97%

100%

88%

85%

82%
Emu Video win rate

Text Faithfulness

60%

97%

81%

79%

80%

92%

91%

99%

96%

97%

87%

56%

40%

2022

Pika Labs

Reuse and Di use

Align Your Latents

PYOCO

Gen-2

CogVideo

Make-A-Video

0%

Imagen Video

20%

2023

Figure 2.5.6

Chapter 2 Preview

39Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.6 Reasoning

Reasoning in AI involves the ability of AI systems to draw logically valid conclusions from different forms of information.
AI systems are increasingly being tested in diverse reasoning contexts, including visual (reasoning about images), moral(understanding moral dilemmas), and social reasoning (navigating social situations).10

2.6 Reasoning
General Reasoning

Massive Multi-discipline Multimodal Understanding

General reasoning pertains to AI systems being

and Reasoning Benchmark for Expert AGI. MMMU

able to reason across broad, rather than specific,

comprises about 11,500 college-level questions

domains. As part of a general reasoning challenge,

from six core disciplines: art and design, business,for example, an AI system might be asked to reason

science, health and medicine, humanities and social

across multiple subjects rather than perform one

science, and technology and engineering (Figure

narrow task (e.g., playing chess).

2.6.1). The question formats include charts, maps,

MMMU: A Massive Multi-discipline
Multimodal Understanding and Reasoning
Benchmark for Expert AGI
In recent years, the reasoning abilities of AI systems
have advanced so much that traditional benchmarks
like SQuAD (for textual reasoning) and VQA (forvisual reasoning) have become saturated, indicatinga need for more challenging reasoning tests.

tables, chemical structures, and more. MMMU is
one of the most demanding tests of perception,knowledge, and reasoning in AI to date. As ofJanuary 2024, the highest performing model isGemini Ultra, which leads in all subject categories
with an overall score of 59.4% (Figure 2.6.2).11 On
most individual task categories, top models are still
well beyond medium-level human experts (Figure
2.6.3). This relatively low score is evidence ofResponding to this, researchers from the United

MMMU’s effectiveness as a benchmark for assessing

States and Canada recently developed MMMU, theAI reasoning capabilities.

10 Some abilities highlighted in the previous sections implicitly involve some form of reasoning. This section highlights tasks that have a more specific reasoning focus.
11 The AI Index reports results from the MMMU validation set, as recommended by the paper authors for the most comprehensive coverage. According to the authors, the test set, with its
unreleased labels and larger size, presents a more challenging yet unbiased benchmark for model performance, ensuring a more robust evaluation. The test set results are available on the
MMMU page.

Chapter 2 Preview

40Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.6 Reasoning

Sample MMMU questions
Source: Yue et al., 2023

Figure 2.6.1

Chapter 2 Preview

41

Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

MMMU: overall accuracy

Source: MMMU, 2023 | Chart: 2024 AI Index report

100%
82.60%, Human expert (medium)51.60%

51.40%

51.10%
45.20%

43.00%

41.20%

41.10%

OmniLMM-12B*

56.80%

Marco-VL*

59.40%

InternLM-XComposer2-VL*

60%

Qwen-VL-PLUS*

Overall accuracy (%)

80%

40%

39.40%In MM-Zephyr-7B*

LLaVA-1.6-34B*

Qwen-VL-MAX*InternVL-Chat-V1.2*

GPT-4V(ision) (Playground)0%

Gemini Ultra*

20%

Figure 2.6.212

Model

MMMU: subject-speci c accuracy
Source: MMMU, 2023 | Table: 2024 AI Index report

MMMU task
category

Leading model

Art and Design

Score

Human expert(medium)

Qwen-VL-MAX*

51.4

84.2

Business

GPT-4V(ision)(Playground)

59.3

86

Science

GPT-4V(ision)(Playground)

54.7

84.7

Health andMedicine

Gemini Ultra*

67.3

78.8

Humanities andSocial Sciences

Gemini Ultra*

78.3

85

Technology andEngineering

Gemini Ultra*

47.1

79.1

Figure 2.6.3

12 An asterisk (*) next to the model names indicates that the results were provided by the authors.

Chapter 2 Preview

42Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

GPQA: A Graduate-Level Google-Proof Q&A
Benchmarkwere crafted by subject-matter experts in various

In the last year, researchers from NYU, Anthropic, and2.6.4). PhD-level experts achieved a 65% accuracy rate

Meta introduced the GPQA benchmark to test generalin their respective domains on GPQA, while nonexpert

multisubject AI reasoning. This dataset consists ofhumans scored around 34%. The best-performing AI

448 difficult multiple-choice questions that cannot bemodel, GPT-4, only reached a score of 41.0% on theeasily answered by Google searching. The questions

main test set (Figure 2.6.5).

fields like biology, physics, and chemistry (Figure

A sample chemistry question from GPQA
Source: Rein et al., 2023

Figure 2.6.4

GPQA: accuracy on the main set
Source: Rein et al., 2023 | Chart: 2024 AI Index report

100%

80%
Accuracy (%) on the main set

72.50%, Expert human validators

60%

40%

39.70%

41.00%

Few-Shot CoT
GPT-4

GPT-4 with search(backo to CoT on abstention)

30.50%, Nonexpert human validators
29.10%

28.00%

Few-Shot CoT
Llama-2-70B-chatFew-Shot CoT
GPT-3.5-turbo-16k

20%

0%

Evaluation method and model

Chapter 2 Preview

Figure 2.6.5

43Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Comparing Humans, GPT-4, and GPT-4V
on Abstraction and Reasoning Tasks

A sample ARC reasoning task
Source: Mitchell et al., 2023

Abstract reasoning involves using known information to solve
unfamiliar and novel problems and is a key aspect of human cognition
that is evident even in toddlers. While recent LLMs like GPT-4 haveshown impressive performance, their capability for true abstract
reasoning remains a hotly debated subject.13 To further explore this
topic, researchers from the Santa Fe Institute tested GPT-4 on theConceptARC benchmark, a collection of analogy puzzles designed
to assess general abstract reasoning skills (Figure 2.6.6). The study
revealed that GPT-4 significantly trails behind humans in abstract
reasoning abilities: While humans score 95% on the benchmark, thebest GPT-4 system only scores 69% (Figure 2.6.7). The developmentof truly general AI requires abstract reasoning capabilities. Therefore,
it will be important to continue tracking progress in this area.

Figure 2.6.6

ConceptARC: accuracy on minimal tasks over all concepts
Source: Mitchell et al., 2023 | Chart: 2024 AI Index report

100%95%, Human performance

80%
69%

Accuracy (%)

65%
60%

40%

25%

23%

20%

0%

GPT-4 Temp = 0

GPT-4 Temp = 0.5

GPT-4V Zero-ShotModel

13 Some claim these models exhibit such reasoning capabilities, while others claim they do not.

Chapter 2 Preview

GPT-4V One-Shot

Figure 2.6.7

44

Artificial IntelligenceIndex Report 2024

Mathematical Reasoning
Mathematical problem-solving benchmarks evaluate
AI systems’ ability to reason mathematically. AI
models can be tested with a range of math problems,from grade-school level to competition-standard
mathematics.

Chapter 2: Technical Performance
2.6 Reasoningthat AI models develop multistep solutions utilizing
arithmetic operations (Figure 2.6.8). GSM8K has
quickly become a favored benchmark for evaluating
advanced LLMs. The top-performing model on GSM8K
is a GPT-4 variant (GPT-4 Code Interpreter), whichscores an accuracy of 97%, a 4.4% improvement from

GSM8K

the state-of-the-art score in the previous year and a

GSM8K, a dataset comprising approximately 8,000

30.4% improvement from 2022 when the benchmark

varied grade school math word problems, requires

was first introduced (Figure 2.6.9).

Sample problems from GSM8K
Source: Cobbe et al., 2023

Figure 2.6.8

Chapter 2 Preview

45Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.6 Reasoning

GSM8K: accuracy

Source: Papers With Code, 2023 | Chart: 2024 AI Index report97%

Accuracy (%)

96%

93%

2022

2023
Figure 2.6.9

Chapter 2 Preview

46

Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

MATH

A sample problem from the MATH dataset

MATH is a dataset of 12,500 challenging

Source: Hendrycks et al., 2023

competition-level mathematics problems
introduced by UC Berkeley researchers in 2021
(Figure 2.6.10). AI systems struggled on MATH
when it was first released, managing to solve only
6.9% of the problems. Performance has significantly
improved. In 2023, a GPT-4-based model posted
the top result, successfully solving 84.3% of thedataset’s problems (Figure 2.6.11).

MATH word problem-solving: accuracy

Figure 2.6.10

Source: Papers With Code, 2023 | Chart: 2024 AI Index report

90%, Human baseline
84.30%

Accuracy (%)

80%

60%

40%

20%

0%

2021

2022

2023
Figure 2.6.11

Chapter 2 Preview

47

Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

PlanBenchtested I-GPT-3 and GPT-4 on 600 problems in the

A planning system receives a specified goal, an

Blocksworld domain (where a hand tries to constructinitial state, and a collection of actions. Each action

stacks of blocks when it is only allowed to move one

is defined by preconditions, which must be met for

block at a time to the table or to the top of a clear

the action to be executed, and the effects that result

block) using one-shot learning and showed that GPT-4

from the action’s execution. The system constructs

could generate correct plans and cost-optimal plansa plan, comprising a series of actions, to achieve theabout 34% of the time, and I-GPT-3 about 6% (Figure

goal from the initial state.

2.6.12). Verifying the correctness of a plan is easier.

Claims have been made that LLMs can solve
planning problems. A group from Arizona State
University has proposed PlanBench, a benchmark
suite containing problems used in the automatedplanning community, especially those used in
the International Planning Competition. They

GPT-4 vs. I-GPT-3 on PlanBench
Source: Valmeekam, 2023 | Table: 2024 AI Index report

Task
Plan generation
Cost-optimal planning
Plan veri cation

GPT-4(instances correct)

I-GPT-3(instances correct)

34.30%

6.80%

33%

5.80%

58.60%

12%

Figure 2.6.12

Chapter 2 Preview

48Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Visual Reasoning

behind their answers (Figure 2.6.13). Performance

Visual reasoning tests how well AI systems canin VCR is measured using the Q->AR score, which

reason across both visual and textual data.

evaluates the machine’s ability to both select the

Visual Commonsense Reasoning (VCR)

correct answer to a question (Q->A) and choose theIntroduced in 2019, the Visual Commonsense

appropriate rationale behind that answer (Q->R). While

Reasoning (VCR) challenge tests the commonsenseAI systems have yet to outperform humans on thisvisual reasoning abilities of AI systems. In thistask, their capabilities are steadily improving. Between

challenge, AI systems not only answer questions

2022 and 2023, there was a 7.93% increase in AI

based on images but also reason about the logic

performance on the VCR challenge (Figure 2.6.14).

A sample question from the Visual Commonsense Reasoning (VCR) challenge
Source: Zellers et al., 2018

Figure 2.6.13

Visual Commonsense Reasoning (VCR) task: Q->AR score
Source: VCR Leaderboard, 2023 | Chart: 2024 AI Index report

85, Human baseline
81.60

80

Q->AR score

70

60

50

2018

2019

2020

2021

2022

2023

Figure 2.6.14

Chapter 2 Preview

49Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Moral Reasoning

A moral story from MoCa
Source: Nie et al., 2023

In the future, AI will be increasingly applied todomains where ethical considerations are crucial,
such as in healthcare and judicial systems. Therefore,
it is essential for AI systems to possess robust moral
reasoning capabilities, enabling them to effectively
navigate and reason about ethical principles andmoral considerations.

MoCa
The ability of AI models to reason in linguistic andvisual domains is well established, yet their capacity
for moral reasoning, especially moral reasoningthat aligns with human moral judgments, is lessunderstood.14 To further explore this topic, a team of
Stanford researchers created a new dataset (MoCa)of human stories with moral elements (Figure 2.6.15).
The researchers then presented these models withstories of human actions and prompted the models
to respond, measuring moral agreement with the

Figure 2.6.15

discrete agreement metric: A higher score indicatescloser alignment with human moral judgment. The
study yielded intriguing results. No model perfectly
matches human moral systems, but newer, larger
models like GPT-4 and Claude show greateralignment with human moral sentiments than smaller
models like GPT-3, suggesting that as AI models
scale, they are gradually becoming more morallyaligned with humans. Of all models surveyed, GPT-4
showed the greatest agreement with human moral
sentiments (Figure 2.6.16).

14 The topic of AI and moral alignment is contentious, as there are no universally agreed-upon moral principles. What constitutes moral alignment for one party may significantly
differ for another.

Chapter 2 Preview

50Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Zero-shot alignment with human judgments on the moral permissibility task: discrete agreement
Source: Nie et al., 2023 | Chart: 2024 AI Index report

40.30

41.90

26.60

26.60

26.60

32.30

32.30

Alpaca-7B

30

GPT-3.5-davinci-v2

37.10

26.60

23.40
20

20.20

18.50

2019

2020

2022

GPT-4

GPT-3.5-turbo

Anthropic-claude-v1

GPT-3.5-davinci-v3

GPT-3-curie-v1

Electra-gen-large

GPT-3-babbage-v1

GPT-2-XL

0

RoBERTa-large

10

ALBERT-xxlarge

Discrete agreement (agg.)

40

2023

Figure 2.6.16

Chapter 2 Preview

51

Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Causal Reasoning

human evaluators as superior to existing ToM

Causal reasoning assesses an AI system’s ability tobenchmarks. BigToM tests LLMs on forward belief

understand cause-and-effect relationships. As AI(predicting future events), forward action (acting

becomes increasingly ubiquitous, it has become

based on future event predictions), and backward

important to evaluate whether AI models can notbelief (retroactively inferring causes of actions)only explain their outputs but also update their(Figure 2.6.17).

conclusions—key aspects of causal reasoning.

In tests of LLMs on the benchmark, GPT-4 was the

BigToM

top performer, with ToM capabilities nearing but not

Assessing whether LLMs have theory-of-mind (ToM)

surpassing human levels (Figure 2.6.18, Figure 2.6.19,

capabilities—understanding and attributing mental

and Figure 2.6.20). More specifically, as measured bystates such as beliefs, intentions, and emotions—has

accuracy in correctly inferring beliefs, GPT-4 closely

traditionally challenged AI researchers. Earlier methods

matched human performance in forward belief andto evaluate ToM in LLMs were inadequate and lacked

backward belief tasks and slightly surpassed humans

robustness. To tackle this problem, in 2023 researchers

in forward action tasks. Importantly, the study

developed a new benchmark called BigToM, designed

shows that LLM performance on ToM benchmarks

for evaluating the social and causal reasoning abilities

is trending upward, with newer models like GPT-of LLMs. BigToM, comprising 25 controls and 5,000

4 outperforming predecessors such as GPT-3.5

model-generated evaluations, has been rated by(released in 2022).

Sample BigToM scenario
Source: Gandhi et al., 2023

Figure 2.6.17

Chapter 2 Preview

52Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Forward action inference with initial belief: accuracy
Source: Gandhi et al., 2023 | Chart: 2024 AI Index report

100%

90%

2022

2023

10% 7%

12% 9%

2022

Figure 2.6.18

35%34%

40%

24%

15%

Human performance

0%

30%

29%

GPT-4-0314

ClaudeGPT-4-0314

Claude-2

Human performance

LLaMA-65

20%

GPT-3.5-turbo

text-davinci-003

40%

LLaMA-65

20%

60%

Claude-2

29%

20%
0%

42%

Claude

47%

73%
62%

text-davinci-003

60%
40%

81%

80%

73%

46%

TB∧FB

GPT-3.5-turbo

76%

99%

98%

96%

93%

80%
Accuracy (%)

TB

97%

100% 96%

Source: Gandhi et al., 2023 | Chart: 2024 AI Index report

TB∧FB

Accuracy (%)

TB

Backward belief inference with initial belief: accuracy

2023

Figure 2.6.19

Forward belief inference with initial belief: accuracy
Source: Gandhi et al., 2023 | Chart: 2024 AI Index report

TB

TB∧FB

100%

91% 90%99%98%

60%

55%52%

40%

35%
27% 25%62%59%

43% 41%31%

2022

Human performance

GPT-4-0314

ClaudeClaude-2

GPT-3.5-turbo

0%

LLaMA-65

20%

text-davinci-003

Accuracy (%)

80%

2023

Figure 2.6.20

Chapter 2 Preview

53Chapter 2: Technical Performance
2.6 Reasoning

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Tübingen Cause-Effect Pairs
Researchers from Microsoft

year’s best by 13 percentage points (Figure 2.6.22). Notably,

and the University of Chicago

GPT-4 outperformed prior covariance-based AI models, whichhave demonstrated that LLMs

were explicitly trained for causal reasoning tasks. Furthermore,are effective causal reasoners.

the researchers discovered that certain prompts, especially those

The team evaluated several

designed to encourage helpfulness, can significantly enhance an

recent LLMs, including GPT-

LLM’s causal reasoning capabilities.

4, using the Tübingen cause-

Sample cause-effect pairs from the Tübingen dataset

effect pairs dataset. This

Source: Kiciman et al., 2023

benchmark comprises over 100
cause-and-effect pairs across
37 subdisciplines, testing AI
systems’ ability to discern causal
relationships (Figure 2.6.21). GPT4’s performance, a 96% accuracy
score, surpassed the previous
Figure 2.6.21

Performance on the Tübingen cause-e ect pairs dataset: accuracy
Source: Kıcıman et al., 2023 | Chart: 2024 AI Index report

100%

96%

75%

75%

79%

81%

GPT-3.5-turbo

83%
80%

text-davinci-002

GPT version
82%

86%

89%
83%

48%

49%

50%

50%

50%

50%

51%

51%

text-ada-001

adatext-babbage-001

text-curie-001

text-davinci-001

babbage

curie

60%

davinci

Accuracy (%)

68%

40%

2021

GPT-3.5-turbo(single prompt)

GPT-3.5-turbo(causal agent)

GPT-4(single prompt)

2020

LMPrior2019

text-davinci-003

2017

Mosaic

2012

bQCD

Slope

0%

PNL-MLP

20%

2022

2023

Figure 2.6.22

Chapter 2 Preview

54Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.7 Audio

AI systems are adept at processing human speech, with audio capabilities that include transcribing spoken words totext and recognizing individual speakers. More recently, AI has advanced in generating synthetic audio content.

2.7 Audio
Generation
2023 marked a significant year in the field of audio

This advancement was highlighted by the releasegeneration, which involves creating synthetic audioof several prominent audio generators, such as

content, ranging from human speech to music files.

UniAudio, MusicGen, and MusicLM.

Chapter 2 Preview

55

Chapter 2: Technical Performance
2.7 Audio

Artificial IntelligenceIndex Report 2024

Highlighted Research:

UniAudio
UniAudio is a high-level language modeling

UniAudio surpasses leading methods in tasks,

technique to create audio content. UniAudio

including text-to-speech, speech enhancement,

uniformly tokenizes all audio types and, like modern

and voice conversion (Figure 2.7.1). With 1 billion

LLMs, employs next-token prediction for high-parameters and trained on 165,000 hours of audio,

quality audio generation. UniAudio is capable of

UniAudio exemplifies the efficacy of big data andgenerating high-quality speech, sound, and music.

self-supervision for music generation.

UniAudio vs. selected prior works in the training stage: objective evaluation metrics
Source: Yang et al., 2023 | Chart: 2024 AI Index report

SIM ↑

WER ↓

PESQ ↑VISQOL ↑DNSMOS ↑

FAD ↓

KL ↓

Text-to-speech

Voice conversion

5

5

4

4

3

2.00

2

0.71

0.62

0

Shen et al. (2023)

0

UniAudio

0.87

0.82

1

Wang et al. (2023e)

Objective evaluation metrics

Speech enhancement
4
2.72

3

3.66
2.63

3

2.44

2

1

1
Richter et al. (2023)

UniAudio

3.96

4

2

0

0

3.35
2.41

2.36
1.88

Wang et al. (2018)

Text-to-sound
5

3

5

2.60

3.12

2.60

1

1
UniAudio0
Model

Chapter 2 Preview

3.65

3
2

Liu et al. (2023a)

UniAudio

4.52

4

2

0

1.68

Text-to-music

4.93

4

UniAudioTarget speaker extraction

3.29

3.21

4.80

3

2.30

2
1

4.90

1.40

Copet et al. (2023)

1.90

UniAudio

Figure 2.7.1

56

Chapter 2: Technical Performance
2.7 Audio

Artificial IntelligenceIndex Report 2024

Highlighted Research:

MusicGEN and MusicLM
Meta’s MusicGen is a novel audio generation

Human evaluators also favor MusicGen for its

model that also leverages the transformer

overall quality (OVL).

architecture common in language models togenerate audio. MusicGen enables users tospecify text for a desired audio outcome andthen fine-tune it using specific melodies. In
comparative studies, MusicGen outshines otherpopular text-to-music models like Riffusion,Moûsai, and MusicLM across various generative
music metrics. It boasts a lower FAD score,
indicating more plausible music generation,a lower KL score for better alignment withreference music, and a higher CLAP score,reflecting greater adherence to textual
descriptions of reference music (Figure 2.7.2).

Chapter 2 Preview

Although MusicGen outperforms certain textto-music models released earlier in the year,
MusicLM is worth highlighting because its release
was accompanied by the launch of MusicCaps, astate-of-the-art dataset of 5.5K music-text pairs.
MusicCaps was used by MusicGen researchers
to benchmark the performance of their familyof models. The emergence of new models like
MusicGen, and new music-to-text benchmarks like
MusicCaps, highlights the expansion of generative
AI beyond language and images into more diverse
skill modalities like audio generation.

57

0.00

Chapter 2 Preview
0.19
OVL.↑

0.31

0.10

0.40
0.32

0.28

0.23
40

20

0
80.51
80.74

81.3

1.22

MusicGen w/o
melody (3.3B)

2.10
1.23

MusicGen w/o
melody (1.5B)

1
1.28

MusicGen w/o
melody (300M)

1.31

MusicGen w/o
melody (3.3B)

79.31

MusicGen w/
random melody (1.5B)

78.43

MusicGen w/
random melody (1.5B)

60
76.11

MusicGen w/o
melody (1.5B)

80

MusicLM100

Ri usion0
Mousai

3.10MusicGen w/o
melody (300M)

7.50

Ri usion

10
KL ↓

14.80

Mousai

3.40

Noise2Music

MusicGen w/o
melody (300M)

Mousai (2023)

MusicGen w/o
melody (1.5B)

0.31
MusicGen w/o
melody (1.5B)

3.80

MusicGen w/o
melody (3.3B)

0.30
4.00

MusicGen w/o
melody (3.3B)

5.00

MusicGen w/o
melody (300M)

5MusicLMMusicGen w/
random melody (1.5B)

MousaiMusicGen (2023)

MusicGen w/
random melody (1.5B)

0.20

Mousai0
Ri usion

FAD VGG ↓15

Ri usion

CLAP SCR ↑Artificial IntelligenceIndex Report 2024
Chapter 2: Technical Performance
2.7 Audio

Highlighted Research:

MusicGEN and MusicLM (cont’d)

Source: Copet et al., 2023 | Chart: 2024 AI Index report

Evaluation of MusicGen and baseline models on MusicCaps
Ri usion (2022)

2
2.06
1.59

84.81

Figure 2.7.2

58

Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.8 Agents

AI agents, autonomous or semiautonomous systems designed to operate within specific environments to accomplish
goals, represent an exciting frontier in AI research. These agents have a diverse range of potential applications, fromassisting in academic research and scheduling meetings to facilitating online shopping and vacation booking.

2.8 Agents
General Agentsassessed over 25 LLM-based agents, including thoseThis section highlights benchmarks and researchbuilt on OpenAI’s GPT-4, Anthropic’s Claude 2, andinto agents that can flexibly operate in general task

Meta’s Llama 2. GPT-4 emerged as the top performer,

environments.

achieving an overall score of 4.01, significantly

AgentBenchAgentBench, a new benchmark designed forevaluating LLM-based agents, encompasses eight
distinct interactive settings, including web browsing,
online shopping, household management, puzzles,
and digital card games (Figure 2.8.1). The study

higher than Claude 2’s score of 2.49 (Figure 2.8.2).
The research also suggests that LLMs released in
2023 outperform earlier versions in agentic settings.
Additionally, the AgentBench team speculated thatagents’ struggles on certain benchmark subsections
can be attributed to their limited abilities in long-term
reasoning, decision-making, and instruction-following.

Description of the AgentBench benchmark
Source: Liu et al., 2023

Figure 2.8.1

Chapter 2 Preview

59Chapter 2: Technical Performance
2.8 Agents

Artificial IntelligenceIndex Report 2024

AgentBench across eight environments: overall score
Source: Liu et al., 2023 | Chart: 2024 AI Index report

4.01

4.00
3.50

2.00

2.49

1.60
1.39

1.00

0.93

0.96

CodeLlama-34b

1.25

Vicuna-13b

1.50

1.71

2.44

Claude-2

2.32

Claude

2.50

2022

GPT-4

Claude-instantchat-bison-001

GPT-3.5-turbo

0.00

text-davinci-003

0.50

text-davinci-002

Overall score

3.00

2023
Figure 2.8.2

Chapter 2 Preview

60Chapter 2: Technical Performance
2.8 Agents

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Voyageur
Recent research by Nvidia, Caltech, UT Austin,

playground for its players (Figure 2.8.3). Voyager

Stanford, and UW Madison demonstrates thatexcels in this environment, adeptly remembering

existing LLMs like GPT-4 can be used to develop

plans, adapting to new settings, and transferring

flexible agents capable of continuous learning.

knowledge. It significantly outperforms previous

The team created Voyager, a GPT-4-based agent

models, collecting 3.3 times more unique items,for Minecraft—a complex video game with notraveling 2.3 times further, and reaching key

set endpoint that is essentially a boundless virtual

milestones 15.3 times faster (Figure 2.8.4).

Voyager in action
Source: Wang et al., 2023

Figure 2.8.3

Voyager’s performance improvements over priorstate of the art in Minecraft
Source: Wang et al., 2023 | Chart: 2024 AI Index report

Voyager’s improvement factor

16

15.30

The launch of Voyager is significant, as AI
researchers have long faced challenges increating agents that can explore, plan, and14

learn in open-ended worlds. While previous AI

12systems like AlphaZero succeeded in closed,
rule-defined environments like chess, Go,10

and shogi, they struggled in more dynamic

8

settings, lacking the ability to continuously

6learn. Voyager, however, demonstrates

4

remarkable proficiency in a dynamic video

3.30

game setting, thereby representing a notable

2.30
2
0

advancement in the field of agentic AI.
Unique items obtained

Distance traveled
Category

Chapter 2 Preview

Speed of unlocking key
tech tree milestones

Figure 2.8.4

61

Chapter 2: Technical Performance
2.8 Agents

Artificial IntelligenceIndex Report 2024

Task-Specific Agentsacross 15 varied research tasks. Examples of the tasks
include improving a baseline model on the CIFAR-10This section highlights benchmarks and research

image dataset and training a language model on

into agents that are optimized to perform in specificover 10 million words in BabyLM. Various LLM-basedtask environments, such as mathematical problem-

agents, including GPT-4, Claude-1, AutoGPT, andsolving or academic research.

LangChain, were tested. The results demonstrate

MLAgentBenchthat although there is promise in AI research agents,

MLAgentBench, a new benchmark for evaluating

performance varies significantly across tasks. While

AI research agents’ performance, tests whether

some agents achieved over 80% on tasks like ogbn-

AI agents are capable of engaging in scientificarxiv (improving a baseline paper classification

experimentation. More specifically, MLAgentBenchmodel), all scored 0% on BabyLM (training a small

assesses AI systems’ potential as computer sciencelanguage model) (Figure 2.8.5). Among these, GPT-4

research assistants, evaluating their performance

consistently delivered the best results.

MLAgentBench evaluation: success rate of select models across tasks
Source: Huang et al., 2023 | Chart: 2024 AI Index report

GPT-4

GPT-4 (no retrieval)

Claude-1

Claude-1 (no retrieval)

AutoGPT

LangChain (React)

Baseline

100%

Success rate

80%

60%

40%

20%

0%

Ba
by
L

M

CL

RS

CI

FA
R-1
0

fat
h

om

fee

ne

t

db

ac

k

ho
us
e-p

ide
ric
e

nti

Task

im

fy-

co
n

db

tra
i

ls

lla

ma

og

-in

bn

fer
e

nc
e

-ar

xiv

Pa
rk

sp
ve
cto
ac
ins
esh
riz
on
ati
ip’s-d
on
tita
ise
n
i
ase
c
Figure 2.8.5

15 The full tasks include: (1) CIFAR-10 (improve a baseline image classification model), (2) imdb (improve a baseline sentiment classification model), (3) ogbn-arxiv (improve a baseline
paper classification model from scratch), (4) house prices (train a regression model), (5) spaceship titanic (train a classifier model from scratch), (6) Parkinson’s-disease (train a time-series
regression model), (7) FathomNet (train an out-of-distribution image classification model), (8) feedback (train an out-of-distribution text regression model), (9) identify contrails (train an
out-of-distribution image segmentation model), (10) CLRS (model classic algorithms over graphs and lists), (11) BabyLM (train language model over 10M words), (12) llama-inference (improve
the runtime/autoregressive generation speed of Llama 7B, (13) vectorization (improve the inference speed of a model), (14) literature-review-tool (perform literature review), and (15) bibtexgeneration (generate BibTex from sketch).

Chapter 2 Preview

62Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.9 Robotics

Over time, AI has become increasingly integrated into robotics, enhancing robots’ capabilities to perform complex
tasks. Especially with the rise of foundation models, this integration allows robots to iteratively learn from theirsurroundings, adapt flexibly to new settings, and make autonomous decisions.

2.9 Robotics
Highlighted Research:

PaLM-E
PaLM-E is a new AI model from Google that

outperforms previous state-of-the-art methods likemerges robotics with language modeling to

SayCan and PaLI on both embodied visual question

address real-world tasks like robotic manipulation

answering and planning (Figure 2.9.2).16 On

and knowledge tasks like question answering androbotic manipulation tasks, PaLM-E outperforms

image captioning. Leveraging transformer-based

competing models (PaLI and CLIP-FT) in its ability

architectures, the largest PaLM-E model is scaled

to detect failures, which is a crucial step for robots

up to 562B parameters. The model is trained

to perform closed-loop planning (Figure 2.9.3).

on diverse visual language as well as robotics
data, which results in superior performance ona variety of robotic benchmarks. PaLM-E also
sets new standards in visual tasks like OK-VQA,
excels in other language tasks, and can engage inchain-of-thought, mathematical, and multi-image
reasoning, even without specific training in theseareas. Figure 2.9.1 illustrates some of the tasks that
the PaLM-E model can perform.
On Task and Motion Planning (TAMP) domains,

PaLM-E is significant in that it demonstrates that
language modeling techniques as well as text
data can enhance the performance of AI systems
in nonlanguage domains, like robotics. PaLM-E
also highlights how there are already linguistically
adept robots capable of real-world interaction andhigh-level reasoning. Developing these kinds ofmultifaceted robots is an essential step in creating
more general robotic assistants that can, forexample, assist in household work.

where robots have to manipulate objects, PaLM-E

16 Embodied Visual Question Answering (Embodied VQA) is a task where agents need to navigate through 3D environments and answer questions about the objects theyvisually perceive in the environment.

Chapter 2 Preview

63Chapter 2: Technical Performance
2.9 Robotics

Artificial IntelligenceIndex Report 2024

Highlighted Research:

PaLM-E (cont’d)
PaLM-E in action
Source: Robotics at Google, 2023

Figure 2.9.1

Performance of select models on TAMP environment: success rate
Source: Driess et al., 2023 | Table: 2024 AI Index report

Model

Embodied VQA q1

Embodied VQA q2

Embodied VQA q3

0

0

98.2

100

Embodied VQA q4

SayCan (oracle a ordances)
PaLI (zero-shot)
99.7

PaLM-E OSRT w/ input
encoding

93.7

Planning p1

Planning p2

38.7

33.3

82.5

76.2

Figure 2.9.2

Select models on mobile manipulation environment
tests: failure detection
Source: Driess et al., 2023 | Table: 2024 AI Index report

Baselines

Failure detection

PaLI (zero-shot)

0.73

CLIP-FT

0.65

CLIP-FT-hindsight

0.89

PaLM-E-12B

0.91
Figure 2.9.3

Chapter 2 Preview

64Chapter 2: Technical Performance
2.9 Robotics

Artificial IntelligenceIndex Report 2024

Highlighted Research:

RT-2
Real-world robots could benefit from certain

and adaptable approaches for conditioning robotic

capabilities possessed by LLMs, such as text andpolicy. It outshines state-of-the-art models likecode generation, as well as visual understanding.

Manipulation of Open-World Objects (MOO) across

RT-2, a new robot released from DeepMind,various benchmarks, particularly in tasks involving

represents an ambitious attempt to create a

unseen objects. On such tasks, an RT-2/PaLM-E

generalizable robotic model that has certain

variant achieves an 80% success rate, significantly

LLM capabilities. RT-2 uses a transformer-based

higher than MOO’s (53%) (Figure 2.9.4). In unseen

architecture and is trained on both robotic

object tasks, RT-2 surpasses the previous year’s

trajectory data that is tokenized into text andstate-of-the-art model, RT-1, by 43 percentage

extensive visual-language data.

points. This indicates an improvement in robotic
performance in novel environments over time.

RT-2 stands out as one of the most impressive

Evaluation of RT-2 models and baselines on seen and unseen tasks: success rate
Source: Brohan et al., 2023 | Chart: 2024 AI Index report

R3M (2022)

100%

VC-1 (2023)

93%

92%

RT-1 (2022)

RT-2-PaLM-E-12B1 (2023)

RT-2-PaLI-X-55B (2023)

91%
80%

80%

75%

73% 72%66%

63%

Success rate

MOO (2023)

60%

62%62%

53%

49%

45%

40%

40%40%

37%

35%

23%22%

35%

20%

20%
11%

0%

32%

11%

8%

12%

10%

1% 0%
Seen tasks

Unseen objects (avg.)

Unseen backgrounds (avg.)
Task

Chapter 2 Preview

Unseen environments (avg.)

Unseen average

Figure 2.9.4

65

Chapter 2: Technical Performance
2.10 Reinforcement Learning

Artificial IntelligenceIndex Report 2024

In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learningfrom their prior actions. Systems are rewarded if they achieve a desired goal and punished if they fail.

2.10 Reinforcement Learning
Reinforcement Learningfrom Human Feedback

seven models reported using RLHF, and in 2023, 16
models reported using RLHF. The rising popularity of
RLHF is also evidenced by the fact that many leading

Reinforcement learning has gained popularity

LLMs report improving their models with RLHF

in enhancing state-of-the-art language models(Figure 2.10.2).

like GPT-4 and Llama 2. Introduced in 2017,
Reinforcement Learning from Human Feedback(RLHF) incorporates human feedback into the

Number of foundation models using RLHF, 2021–23
Source: AI Index, 2024 | Chart: 2024 AI Index report

reward function, enabling models to be trained forcharacteristics like helpfulness and harmlessness.

14

This year, the AI Index tracked data on the number
their training. More specifically, the Index teamlooked through the technical reports and otherdocumentation of all models included in CRFM’s
Ecosystem graph, one of the most comprehensive

12
Number of models

of foundation models using RLHF as part of16

16

repositories of the foundation model ecosystem.17

10
8

7

6
4

Figure 2.10.1 illustrates how many foundation models
2reported using RLHF over time. In 2021, no newly
released foundation models used RLHF. In 2022,

0

0
2021

2022

2023
Figure 2.10.1

RLHF usage among foundation models
Source: AI Index, 2024 | Table: 2024 AI Index report

GPT-4

Llama 2✓Claude-2✓

Gemini✓

Mistral-7B✓

x
Figure 2.10.2

17 It is possible that more models use RLHF as part of their training than reported. The Index only tracks data for models that publicly report using RLHF.

Chapter 2 Preview

66Chapter 2: Technical Performance
2.10 Reinforcement Learning

Artificial IntelligenceIndex Report 2024

Highlighted Research:

RLAIF

RLAIF and RLHF vs. SFT baseline: win rate
Source: Lee et al., 2023 | Chart: 2024 AI Index report

100%
RLHF

RLAIF

RLHF is a powerful method for aligning AI
80%models but can be hindered by the time73%

and labor required to generate human64%

preference datasets for model alignment.
Learning from AI Feedback (RLAIF) uses

63%

60%
Win rate

As an alternative, Reinforcement

71%

40%

reinforcement learning based on thepreferences of LLMs to align other AI
models toward human preferences.

20%

Recent research from Google Researchcompares RLAIF with RLHF, the traditional0%gold standard, to assess whether RLAIF

Summarization

can serve as a reliable substitute. The
study finds that both RLAIF and RLHF
are preferred over supervised fine-tuning

Helpfulness
Task

Harmless rate by policy

Source: Lee et al., 2023 | Chart: 2024 AI Index report

100%(SFT) for summarization and helpfulness

88%

tasks, and that there is not a statisticallysignificant difference in the degree to

80%

76%

which RLHF is preferred (Figure 2.10.3).
Notably, in harmless dialogue generation
harmful outputs, RLAIF (88%) surpasses
RLHF (76%) in effectiveness (Figure
2.10.4). This research indicates that RLAIF

64%
Harmless ratetasks focused on producing the least

Figure 2.10.3

60%

40%

could be a more resource-efficient andcost-effective approach for AI model

20%alignment.
0%

Chapter 2 Preview

SFT

RLHF
PolicyRLAIF

Figure 2.10.4

67

Chapter 2: Technical Performance
2.10 Reinforcement Learning

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Direct Preference Optimization
As illustrated above, RLHF is a useful method

Direct Preference Optimization (DPO). DPO isfor aligning LLMs with human preferences.

simpler than RLHF but equally effective. The

However, RLHF requires substantial computational

researchers show that DPO is as effective as otherresources, involving the training of multiple

existing alignment methods, such as Proximal

language models and integrating LM policy

Policy Optimization (PPO) and Supervised Fine-

sampling within training loops. This complexity

Tuning (SFT), on tasks like summarization (Figure

can hinder its broader adoption.

2.10.5). The emergence of techniques like DPO
suggests that model alignment methods are

In response, researchers from Stanford and CZ

becoming more straightforward and accessible.

Biohub have developed a new reinforcement
learning algorithm for aligning models named

Comparison of di erent algorithms on TL;DR summarization task across di erent sampling temperatures

Source: Rafailov et al., 2023 | Table: 2024 AI Index report

DPO

PPO

Best of 128

SFT

Preferred-FT

GPT-J

0.70

0.60

Win rate

0.50

Human baseline

0.40

0.30

0.20

0.10

0.00
0.00

0.25

0.50

0.75

1.00

Sampling temperature

Figure 2.10.5

Chapter 2 Preview

68Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

This section focuses on research exploring critical properties of LLMs, such as their capacity for sudden behavioral
shifts and self-correction in reasoning. It is important to highlight these studies to develop an understanding of how
LLMs, which are increasingly representative of the frontier of AI research, operate and behave.

2.11 Properties of LLMs
Highlighted Research:

Challenging the Notion of Emergent Behavior
Many papers have argued that LLMs exhibit

multiple-choice grading are used to evaluate

emergent abilities, meaning they can unpredictably

models, emergent abilities seem more apparent.

and suddenly display new capabilities at larger

In contrast, when linear or continuous metrics

scales. This has raised concerns that even larger

are employed, these abilities largely vanish.

models could develop surprising, and perhaps

Analyzing a suite of benchmarks from BIG-

uncontrollable, new abilities.

bench, a comprehensive LLM evaluation tool,18

However, research from Stanford challenges thisnotion, arguing that the perceived emergence of newcapabilities is often a reflection of the benchmarks
used for evaluation rather than an inherent propertyof the models themselves. The researchers foundthat when nonlinear or discontinuous metrics likethe researchers noted emergent abilities on
only five of the 39 benchmarks (Figure 2.11.1).
These findings have important implications for AI
safety and alignment research as they challengea prevailing belief that AI models will inevitablylearn new, unpredictable behaviors as they scale.

18 Some of these papers include Brown et al., 2023, Ganguli et al., 2022, Srivastava et al., 2022, and Wei et al., 2022.

Chapter 2 Preview

69Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.11 Properties of LLMs

Highlighted Research:

Challenging the Notion of Emergent Behavior (cont’d)Emergence score over all Big-bench tasks
Source: Schaeffer et al., 2023

Figure 2.11.1

Chapter 2 Preview

70Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Changes in LLM Performance Over Time
Publicly usable closed-source LLMs, such as GPT-4,

that performance declined on several tasks. For

Claude 2, and Gemini, are often updated over time

instance, the June version of GPT-4, comparedby their developers in response to new data or user

to the March version, was 42 percentage points

feedback. However, there is little research on how

worse at generating code, 16 percentage points

the performance of such models changes, if at all, in

worse at answering sensitive questions, and 33

response to such updating.

percentage points worse on certain mathematical

A study conducted at Stanford and Berkeley
explores the performance of certain publicly usable
LLMs over time and highlights that, in fact, theirperformance can significantly vary. More specifically,
the study compared the March and June 2023
versions of GPT-3.5 and GPT-4 and demonstrated

Chapter 2 Preview

tasks (Figure 2.11.2). The researchers also foundthat GPT-4’s ability to follow instructions
diminished over time, which potentially explains
the broader performance declines. This researchhighlights that LLM performance can evolve overtime and suggests that regular users should be
mindful of such changes.

71

Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Changes in LLM Performance Over Time (cont’d)Performance of the March 2023 and June 2023 versions of GPT-4 on eight tasks
Source: Chen et al., 2023 | Chart: 2024 AI Index report

Math I: Prime vs. Composite (n=1,000)
84%

Math II: Happy Numbers (n=500)
84%
80%

60%

51%

40%

Accuracy

Accuracy

80%

35%

40%
20%

20%
0%

60%

2023-Mar0%2023-Jun

2023-MarAnswering Sensitive Questions (n=100)
100%
21%

10%
5%
0%

2023-MarResponse rate

Response rate

30%

20%

60%
40%
23%

60%

38%
Directly EXE

Exact match

80%

0%

2023-Jun

30%
20%

98%

20%

LangChain HotpotQA Agent (n=7,405)
40%2023-JunOpinionQA Survey (n=1,506)

2023-Mar2023-Jun

Code Generation and Formatting (n=50)
52%

40%

20%
10%

10%

Accuracy80%2023-Mar2023-MarUSMLE Medical Exam (n=340)
87%
82%

40%
20%

2023-Mar2023-Jun

2023-Jun

Visual Reasoning (n=467)
30%

60%

0%

0%

2023-Jun

Exact match

0%

1%

25%

27%

20%
10%
0%

2023-Mar2023-Jun

Figure 2.11.2

Chapter 2 Preview

72Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

Highlighted Research:

LLMs Are Poor Self-Correctors
It is generally understood that LLMs like GPT-4

Researchers from DeepMind and the University

have reasoning limitations and can sometimesof Illinois at Urbana–Champaign tested GPT-4’s

produce hallucinations. One proposed solution

performance on three reasoning benchmarks:

to such issues is self-correction, whereby LLMs

GSM8K (grade-school math), CommonSenseQA

identify and correct their own reasoning flaws. As(common-sense reasoning), and HotpotQA

AI’s societal role grows, the concept of intrinsic(multidocument reasoning). They found that whenself-correction—allowing LLMs to autonomously

the model was left to decide on self-correction

correct their reasoning without external guidance—

without guidance, its performance declined acrossis especially appealing. However, it is currently not

all tested benchmarks (Figure 2.11.3).

well understood whether LLMs are in fact capableof this kind of self-correction.
GPT-4 on reasoning benchmarks with intrinsic self-correction
Source: Huang et al., 2023 | Chart: 2024 AI Index report

Standard prompting: 1 call
100%

Self-correct (round 1): 3 calls

95.50%
91.50%

89.00%
82.00%

80%

Accuracy (%)

Self-correct (round 2): 5 calls

79.50%

80.00%

60%
49.00%

49.00%
43.00%

40%

20%

0%

GSM8K

CommonSenseQA

HotpotQA

Figure 2.11.3

Chapter 2 Preview

73Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

Closed vs. Open Model Performance

innovation, and enhance transparency within the AI

As LLMs become increasingly ubiquitous, debate

ecosystem. Others contend that open-source models

intensifies over their varying degrees of accessibility.

present considerable security risks, such as facilitating

Some models such as Google’s Gemini remain closed,

the creation of disinformation or bioweapons, andaccessible solely to their developers. In contrast,

should therefore be approached with caution.

models like OpenAI’s GPT-4 and Anthropic’s Claude
2 offer limited access, available publicly via an API.

In the context of this debate, it is important to

However, model weights are not fully released, which

acknowledge that current evidence indicates ameans the model cannot be independently modified

notable performance gap between open and closedby the public or further scrutinized. Conversely,models.19 Figures 2.11.4 and 2.11.5 juxtapose theMeta’s Llama 2 and Stability AI’s Stable Diffusion

performances of the top closed versus open model

adopt an open approach, fully releasing their model

on a selection of benchmarks.20 On all selected

weights. Open-source models can be modified andbenchmarks, closed models outperform open ones.

freely used by anyone.

Specifically, on 10 selected benchmarks, closed
models achieved a median performance advantage ofViewpoints differ on the merits of closed versus open

24.2%, with differences ranging from as little as 4.0%

AI models. Some argue in favor of open models, citingon mathematical tasks like GSM8K to as much as

their ability to counteract market concentration, foster

317.7% on agentic tasks like AgentBench.

Score di erentials of top closed vs. open models on select benchmarks
Source: AI Index, 2024 | Table: 2024 AI Index report

BenchmarkTask category

Best closed model score

Best open model score

AgentBenchAgent-based behavior

4.01

0.96

Chatbot Arena Leaderboard

General language

1,252

1,149

GPQA

General reasoning

41.00%

29.10%

GSM8K

Mathematical reasoning

97.00%

93.30%HELM

General language

HumanEval

Coding

MATH
MMLU

0.96

0.82

96.30%

62.20%

Mathematical reasoning

84.30%

60.40%

General language

90.04%

70.60%MMMU

General reasoning

59.40%

51.10%

SWE-bench

Coding

4.80%

3.97%
Figure 2.11.4

19 By closed models, the AI Index is referring both to models that are fully closed and those with limited access.
20 The data in this section was collected in early January 2024.

Chapter 2 Preview

74Chapter 2: Technical Performance
2.11 Properties of LLMs

Artificial IntelligenceIndex Report 2024

Performance of top closed vs. open models on select benchmarks
Source: AI Index, 2024 | Chart: 2024 AI Index report

317.71%

Closed vs. open percentage score di erence

300%

250%

200%

150%

100%
54.82%
50%
17.07%
0%

HELM

39.57%

27.54%

20.91%

8.96%
MMLU
General language

Chatbot Arena
Leaderboard

3.97%
HumanEval

SWE-bench
Coding

MATH

GSM8K

AgentBench

Mathematical reasoning

Agent-based
behavior

BenchmarkFigure 2.11.5

Chapter 2 Preview

75Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Artificial IntelligenceIndex Report 2024

As LLMs use increases, techniques are being sought to enhance their performance and efficiency. This section examinessome of those advances.

2.12 Techniques for LLM Improvement
PromptingMastering the art of crafting effective prompts

Prompting, a vital aspect of the AI pipeline, entails

significantly enhances the performance of LLMs

supplying a model with natural language instructions

without requiring that models undergo underlyingthat describe tasks the model should execute.

improvements.

Highlighted Research:

Graph of Thoughts Prompting
Chain of thought (CoT) and Tree of Thoughts(ToT) are prompting methods that can improve

Graph of Thoughts (GoT) reasoning flow
Source: Besta et al., 2023

the performance of LLMs on reasoning tasks. In
2023, European researchers introduced anotherprompting method, Graph of Thoughts (GoT), thathas also shown promise (Figure 2.12.1). GoT enables
LLMs to model their thoughts in a more flexible,
graph-like structure which more closely mirrors
actual human reasoning. The researchers thendesigned a model architecture to implement GoT
and found that, compared to ToT, it increased thequality of outputs by 62% on a sorting task while
reducing cost by around 31% (Figure 2.12.2).

Figure 2.12.1

Chapter 2 Preview

76Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Graph of Thoughts Prompting (cont’d)Number of errors in sorting tasks with ChatGPT-3.5
Source: Besta et al., 2023 | Chart: 2024 AI Index report

32 elements

64 elements

128 elements

Number of incorrectly sorted elements (mean)

123.93

120

120

120

100

100

100

80

80

80

60

60

60

40

40

20

20

64.56

0

6.58

4.53

CoT

IO

38.12

63.04
52.87

40
21.18

19.53

16.79

17.52

20
7.00

2.25

1.90

1.84

ToT2

ToT

GoT

(L=3, k=10) (L=2, k=20)

0

IO

CoT

ToT2

ToT

(L=7, k=10) (L=4, k=20)

GoT

0

IO

CoT

ToT2

ToT

GoT

(L=10, k=10) (L=4, k=20)

Figure 2.12.2

Chapter 2 Preview

77Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Artificial IntelligenceIndex Report 2024

Highlighted Research:

Optimization by PROmpting (OPRO)A paper from DeepMind has introduced

prompts aim to enhance the performance of AI

Optimization by PROmpting (OPRO), a method

systems on particular benchmarks. Compared tothat uses LLMs to iteratively generate prompts

other prompting approaches like “let’s think stepto improve algorithmic performance. OPRO usesby step” or an empty starting point, ORPO leads

natural language to guide LLMs in creating new

to significantly greater accuracy on virtually all 23

prompts based on problem descriptions andBIG-bench Hard tasks (Figure 2.12.4).

previous solutions (Figure 2.12.3). The generated
Sample OPRO
prompts andoptimization
progress
Source: Yang et al., 2023

Figure 2.12.3

Accuracy di erence on 23 BIG-bench Hard (BBH) tasks using PaLM 2-L scorer

Source: Yang et al., 2023 | Chart: 2024 AI Index report“Let’s think step by step” instruction

Empty instruction
60
50
40

20

30
0

20
10
0

boolean_expressions
causal_judgement
date_understanding
disambiguation_qa
dyck_languages
formal_fallacies
geometric_shapes
hyperbaton
logical_deduction_seven_objects
movie_recommendation
multistep_arithmetic_two
navigate
object_counting
penguins_in_a_tablereasoning_about_colored_objects
ruin_names
salient_translation_error_detection
snarkssports_understanding
temporal_sequences
tracking_shu�ed_objects_seven_objects
web_of_lies
word_sorting−20

boolean_expressions
causal_judgement
date_understanding
disambiguation_qa
dyck_languages
formal_fallacies
geometric_shapes
hyperbaton
logical_deduction_seven_objects
movie_recommendation
multistep_arithmetic_two
navigate
object_counting
penguins_in_a_tablereasoning_about_colored_objects
ruin_names
salient_translation_error_detection
snarkssports_understanding
temporal_sequences
tracking_shu�ed_objects_seven_objects
web_of_lies
word_sorting

Accuracy di�erence

40

Task

Figure 2.12.4

Chapter 2 Preview

78Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Artificial IntelligenceIndex Report 2024

Fine-Tuning

Fine-tuning not only boosts overall model

Fine-tuning has grown increasingly popular as aperformance but also sharpens the model’s

method of enhancing LLMs and involves further

capabilities on specific tasks. It also allows for moretraining or adjusting models on smaller datasets.

precise control over the model’s behavior.

Highlighted Research:

QLoRA manages to increase efficiency with

QLoRA

techniques like a 4-bit NormalFloat (NF4), double
quantization, and page optimizers. QLoRA is

QLoRA, developed by researchers from theused to train a model named Guanaco, which

University of Washington in 2023, is a new method

matched or even surpassed models like ChatGPT

for more efficient model fine-tuning. It dramaticallyin performance on the Vicuna benchmark (areduces memory usage, enabling the fine-tuning

benchmark that ranks the outputs of LLMs) (Figureof a 65 billion parameter model on a single 48

2.12.5). Remarkably, the Guanaco models were

GB GPU while maintaining full 16-bit fine-tuning

created with just 24 hours of fine-tuning on a single

performance. To put this in perspective, fine-tuning

GPU. QLoRa highlights how methods for optimizing

a 65B Llama model, a leading open-source LLM,

and further improving models have become more

typically requires about 780 GB of GPU memory.

efficient, meaning fewer resources will be required

Therefore, QLoRA is nearly 16 times more efficient.

to make increasingly capable models.

Model competitions based on 10,000 simulations using GPT-4 and the Vicuna benchmark
Source: Dettmers et al., 2023 | Chart: 2024 AI Index report

1,400

1,348

1,200

Elo rating (mean)

1,000
879

902

916

Bard

Guanaco 13B

966

974

992

ChatGPT

Vicuna 13B

Guanaco 33B

1,022

800

600

400

200

0

Guanaco 7B

Guanaco 65B

GPT-4

Figure 2.12.5

Chapter 2 Preview

79Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Attention

AI’s broader adoption. Optimization methods aim to

LLMs can flexibly handle various tasks but often

enhance AI’s efficiency by, for example, improving

demand substantial computational resources to train.

memory usage, thereby making LLMs more

As previously noted, high training costs can hinderaccessible and practical.

Highlighted Research:Flash-Decoding
Flash-Decoding, developed by Stanford

inference: For example, on a 256 batch size andresearchers, tackles inefficiency in traditional

256 sequence length, Flash-Decoding is 48 times

LLMs by speeding up the attention mechanism,

faster than PyTorch Eager and six times fasterparticularly in tasks requiring long sequences.

than FlashAttention-2 (Figure 2.12.7). Inference

It achieves this by parallelizing the loading ofon models like ChatGPT can cost $0.01 perkeys and values, then separately rescaling andresponse, which can become highly expensive

combining them to maintain right attention outputs

when deploying such models to millions of users.

(Figure 2.12.6). In various tests, Flash-Decoding

Innovations like Flash-Decoding are critical foroutperforms other leading methods like PyTorch

reducing inference costs in AI.

Eager and FlashAttention-2, showing much fasterFlash-Decoding
operation process
Source: Dao et al., 2023

Figure 2.12.6

Chapter 2 Preview

80Chapter 2: Technical Performance
2.12 Techniques for LLM Improvement

Artificial IntelligenceIndex Report 2024

Highlighted Research:Flash-Decoding (cont’d)Performance comparison of multihead attention algorithms across batch sizes and sequence lengths

Source: Dao et al., 2023 | Chart: 2024 AI Index report

PyTorch Eager (2022)

Flash-Attention v2.0.9 (2023)bsbs

Flash-Decoding (2023)

Run-time (in microseconds)

4,000

3,000

2,000

1,000

0

bs

=2

=12

56
, se

qle

=6

8,
se

n=
2

qle

56

n=
5

12

4,

bs

=3

seq

len
=1,
02
4

2,

bs
seq

bs

len
=2

=16
, se

,04

qle

8

bs

=8
, se

=4

n=
4

qle

,09
6

n=
8,1
92

bs

, se

=2

qle

bs

n=
16,
38

, se

4

=1,

qle

n=
3

2,7
68

bs

seq

=1,

len
=6

5,5
3

6

seq

len
=13
1,0
7

2

Figure 2.12.7

Chapter 2 Preview

81Chapter 2: Technical Performance
2.13 Environmental Impact of AI Systems

Artificial IntelligenceIndex Report 2024

This section examines trends in the environmental impact of AI systems, highlighting the evolving landscape of transparency andawareness. Historically, model developers seldom disclosed the carbon footprint of their AI systems, leaving researchers to maketheir best estimates. Recently, there has been a shift toward greater openness, particularly regarding the carbon costs of training AI
models. However, disclosure of the environmental costs associated with inference—a potentially more significant concern—remains
insufficient. This section presents data on carbon emissions as reported by developers in addition to featuring notable research
exploring the intersection of AI and environmental impact. With AI models growing in size and becoming more widely used, it has
never been more critical for the AI research community to diligently monitor and mitigate the environmental effects of AI systems.

2.13 Environmental Impact of AI Systems
General Environmental Impact

Emission data varies widely. For instance, Meta’s Llama

Trainingof carbon, which is nearly 291 times more than the

2 70B model released approximately 291.2 tonnes

Figure 2.13.1 presents the carbon released by (inemissions released by one traveler on a round-trip flighttonnes) of select LLMs during their training, comparedfrom New York to San Francisco, and roughly 16 times

with human reference points. Emissions data of

the amount of annual carbon emitted by an averagemodels marked with an asterisk were estimated byAmerican in one year.21 However, the emissions fromindependent researchers as they were not disclosed

Llama 2 are still less than the 502 tonnes reportedly

by their developers.

released during the training of OpenAI’s GPT-3.

CO2 equivalent emissions (tonnes) by select machine learning models and real-life examples, 2020–23
Source: AI Index, 2024; Luccioni et al., 2022; Strubell et al., 2019 | Chart: 2024 AI Index report

400

300

200

100

2022

2023

Luminous Base (13B)

Luminous Extended (30B)

Starcoder (15.5B)

Granite (13B)

Llama 2 (7B)

Llama 2 (13B)

Llama 2 (34B)

Llama 2 (70B)

2021

BLOOM (176B)*

2020

OPT (175B)*

Gopher (280B)*

063.00, Car, avg. incl. fuel, 1 lifetime
18.08, American life, avg., 1 year
5.51, Human life, avg., 1 year
0.99, Air travel, 1 passenger, NY–SF

GPT-3 (175B)*

CO2 equivalent emissions (tonnes)

500

Figure 2.13.1

21 In its technical report on Llama 2, Meta notes that it offsets all the carbon emissions generated during the model’s training process.

Chapter 2 Preview

82Chapter 2: Technical Performance
2.13 Environmental Impact of AI Systems

Artificial IntelligenceIndex Report 2024

The variance in emission estimates is due to factors

Super Cluster). However, smaller models can still havesuch as model size, data center energy efficiency, andhigh emissions if trained on energy grids powered by

the carbon intensity of energy grids. Figure 2.13.2 shows

less efficient energy sources. Some estimates suggest

the emissions of select models in relation to their size.

that model emissions have declined over time, which isGenerally, larger models emit more carbon, a trend

presumably tied to increasingly efficient mechanisms ofclearly seen in the Llama 2 model series, which were allmodel training. Figure 2.13.3 features the emissions of

trained on the same supercomputer (Meta’s Research

select models along with their power consumption.

CO2 equivalent emissions (tonnes) and number of parameters by select machine learning models
Source: AI Index, 2024; Luccioni et al., 2022 | Chart: 2024 AI Index report

Gopher (280B)

Number of parameters (log scale)

BLOOM (176B)

OPT (175B)

GPT-3 (175B)

100B
Llama 2 (70B)

Llama 2 (34B)

Luminous Extended (30B)

Starcoder (15.5B)
Granite (13B)

Luminous Base (13B)

Llama 2 (13B)

10B
Llama 2 (7B)

10100
CO2 equivalent emissions (log scale - tonnes)

Figure 2.13.2

Environmental impact of select models

Source: AI Index, 2024; Luccioni et al., 2022 | Table: 2024 AI Index report

Model and number ofparameters

Year

Power
consumption(MWh)C02 equivalentemissions(tonnes)

Gopher (280B)

2021

1,066

BLOOM (176B)

2022

433

352
25

GPT-3 (175B)

2020

1,287

502

OPT (175B)

2022

324

70

Llama 2 (70B)

2023

400

291.42

Llama 2 (34B)

2023

350

153.90

Llama 2 (13B)

2023

400

62.44

Llama 2 (7B)

2023

400

31.22

Granite (13B)

2023

153

22.23

Starcoder (15.5B)

2023

89.67

16.68

Luminous Base (13B)

2023

33

3.17

Luminous Extended (30B)

2023

93

11.95

Chapter 2 Preview

Figure 2.13.3

83

Chapter 2: Technical Performance
2.13 Environmental Impact of AI Systems

Artificial IntelligenceIndex Report 2024

A major challenge in evaluating the environmental

training AI models can be significant. While the per-impacts of AI models is a lack of transparency about

query emissions of inference may be relatively low,emissions. Consistent with findings from other studies,

the total impact can surpass that of training whenmost prominent model developers do not report carbon

models are queried thousands, if not millions, ofemissions, hampering efforts to conduct thorough andtimes daily. Research on the emissions from model

accurate evaluations of this metric. For example, many

inference is scant. A study by Luccioni et al., publishedprominent model developers such as OpenAI, Google,

in 2023, is among the first to comprehensively assess

Anthropic, and Mistral do not report emissions in

the emissions from model inference. Figure 2.13.4

training, although Meta does.

illustrates the emissions from 1,000 inferences across22

various model tasks, revealing that tasks like image

Inference

generation have a much higher carbon footprint thanAs highlighted earlier, the environmental impact oftext classification.

Carbon emissions by task during model inference
Source: Luccioni et al., 2023 | Chart: 2024 AI Index report

Model emissions (g of CO2 - log scale)

1000

100

10

1

Te
xt

cla

ssi

To
ke
n
ca
ti

on

cla

ssi

Qu
est

ion

ca
ti

on

Fil
l
an

Im

ma

sw
eri

sk

ng

ag

ec

las

Ze
ro
si

Ob

-sh

ca
ti

on

jec

ot
sen

Ze
ro

tim

td

-sh

ete

en

ta

cti

na

on

lys

is

Te
xt

ot
q

ue
st

ge

ion

Su
m

ne

an

rat

ion

sw
eri

ng

Im
Ze
ag
roeg
ec
sh
o
ap
en
t
ati
su
tio
era
on
m
nin
tio
ma
g
n
riz
ati
on
Im

ma

riz

ag

Task

Figure 2.13.4

22 Research also suggests that the reporting of carbon emissions on open model development platforms, such as Hugging Face, is declining over time.

Chapter 2 Preview

84Chapter 2: Technical Performance
2.13 Environmental Impact of AI Systems

Artificial IntelligenceIndex Report 2024

Positive Use Cases
Despite the widely recognized environmental costs

efforts.23 These applications include enhancingof training AI systems, AI can contribute positively tothermal energy system management, improving pest

environmental sustainability. Figure 2.13.5 showcases acontrol strategies, and boosting urban air quality.

variety of recent cases where AI supports environmental

Positive AI environmental use cases

Source: Fang et al., 2024 | Table: 2024 AI Index report

Use case

AI contribution

ReferenceManagement of thermal energy storage systems

Anticipating thermal energy needs and managing thermal energy storage
systems.

Olabi et al., 2023

Improving waste management

Saving time and costs in waste-to-energy conversion, waste sorting, andwaste monitoring.

Fang et al., 2023

More e ciently cooling buildings

Optimizing the energy usage associated with air-conditioning.

Luo et al., 2022

Improving pest management

Identifying and eliminating pests in commercial tomato harvests.

Rustia et al., 2022

Enhancing urban air quality

Forecasting and predicting air quality in urban cities.

Shams et al., 2021
Figure 2.13.5

23 Several of the data points in Figure 2.13.5 were adopted from this literature review on the topic of AI and sustainability.

Chapter 2 Preview

85Chapter 2: Technical Performance
Appendix

Artificial IntelligenceIndex Report 2024

AppendixAcknowledgments
The AI Index would like to acknowledge Andrew Shi for
his work doing a literature review on the environmental
impact of AI models; Emily Capstick for her work studying
the use of RLHF in machine learning models; Sukrut Oakfor his work generating sample Midjourney generations;and Emma Williamson for her work identifying significant
AI technical advancements for the timeline.

HEIM, please read the original paper.
8. HELM: Data on HELM was taken from the HELM
leaderboard in January 2024. To learn more about
HELM, please read the original paper.
9. HumanEval: Data on HumanEval was taken fromthe HumanEval Papers With Code leaderboard inJanuary 2024. To learn more about HumanEval,
please read the original paper.
10. M
 ATH: Data on MATH was taken from the MATH

Benchmarks

Papers With Code leaderboard in January 2024.

1. AgentBench: Data on AgentBench was taken fromoriginal paper.

the AgentBench paper in January 2024. To learn more

To learn more about MATH, please read the
11. M LAgentBench: Data on MLAgentBench was

about AgentBench, please read the original paper.

taken from the MLAgentBench paper in January

2. BigToM: Data on BigToM was taken from the BigToM

2024. To learn more about MLAgentBench, please

paper in January 2024. To learn more about BigToM,
please read the original paper.

read the original paper.
12. MMLU: Data on MMLU was taken from the

3. C hatbot Arena Leaderboard: Data on the Chatbot

MMLU Papers With Code leaderboard in January

Arena Leaderboard was taken from the Chatbot

2024. To learn more about MMLU, please read the

Arena Leaderboard in January 2024. To learn more

original paper.

about the Chatbot Arena Leaderboard, please readthe original paper.

13. MMMU: Data on MMMU was taken from the
MMMU leaderboard in January 2024. To learn

4. EditVal: Data on EditVal was taken from the EditVal

more about MMMU, please read the original paper.

paper in January 2024. To learn more about EditVal,

14. MoCa: Data on MoCa was taken from the MoCa

please read the original paper.
5. G
 PQA: Data on GPQA was taken from the GPQA
paper in January 2024. To learn more about GPQA,
please read the original paper.
6. GSM8K: Data on GSM8K was taken from thepaper in January 2024. To learn more about
MoCa, please read the original paper.
15. PlanBench: Data on PlanBench was taken from the
PlanBench paper in January 2024. To learn more
about PlanBench, please read the original paper.

GSM8K Papers With Code leaderboard in January

16. S
 WE-bench: Data on SWE-bench was taken from2024. To learn more about GSM8K, please read thethe SWE-bench leaderboard in January 2024. To

original paper.

learn more about SWE-bench, please read the

7. HEIM: Data on HEIM was taken from the HEIM

original paper.

leaderboard in January 2024. To learn more about

Chapter 2 Preview

86Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

17. TruthfulQA: Data on TruthfulQA was taken fromthe TruthfulQA Papers With Code leaderboard inJanuary 2024. To learn more about TruthfulQA,
please read the original paper.
18. UCF101: Data on UCF101 was taken from the
UCF101 Papers With Code leaderboard in January
2024. To learn more about UCF101, please read theoriginal paper.
19. VCR: Data on VCR was taken from the VCR
leaderboard in January 2024. To learn more about
VCR, please read the original paper.
20. VisIT-Bench: Data on VisIT-Bench was taken fromthe VisIT-Bench leaderboard in January 2024. To
learn more about VisIT-Bench, please read theoriginal paper.

Environmental Impact
To assess the environmental impact of AI models, theAI Index team surveyed technical reports of prominent
foundation models to determine whether the model
developers disclosed carbon emissions. The Index
also reviewed papers by researchers that estimatedthe carbon footprint of various models. The technicalreports surveyed, as well as the papers estimating thecarbon impact of various models, are included in theworks cited for this chapter.

RLHF
To identify foundation models using RLHF, the AI Index
team reviewed the technical documentation of every
foundation model included in the Ecosystem Graph,and searched for evidence that RLHF had been used in
the model’s development process. The year in which amodel is said to have used RLHF refers to the year themodel was released.

Chapter 2 Preview

87Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

Works Cited
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M.,
Sharifi, M., Zeghidour, N. & Frank, C. (2023). MusicLM: Generating Music From Text (arXiv:2301.11325). arXiv.
http://arxiv.org/abs/2301.11325.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog,
A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., … Zeng, A. (2022). Do As I Can, Not As I Say:
Grounding Language in Robotic Affordances (arXiv:2204.01691). arXiv. https://doi.org/10.48550/arXiv.2204.01691.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C.,
Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., … Kaplan, J. (2022). Constitutional
AI: Harmlessness From AI Feedback (arXiv:2212.08073). arXiv. https://doi.org/10.48550/arXiv.2212.08073.
Bairi, R., Sonwane, A., Kanade, A., C, V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B. & Shet, S. (2023). CodePlan:
Repository-Level Coding Using LLMs and Planning (arXiv:2309.12499). arXiv. https://doi.org/10.48550/arXiv.2309.12499.
Basu, S., Saberi, M., Bhardwaj, S., Chegini, A. M., Massiceti, D., Sanjabi, M., Hu, S. X. & Feizi, S. (2023). EditVal: Benchmarking
Diffusion Based Text-Guided Image Editing Methods (arXiv:2310.02426). arXiv. http://arxiv.org/abs/2310.02426.
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski,
H., Nyczyk, P. & Hoefler, T. (2024). Graph of Thoughts: Solving Elaborate Problems with Large Language Models(arXiv:2308.09687). arXiv. http://arxiv.org/abs/2308.09687.
Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R. & Schmidt, L. (2023). VisIT-Bench:
A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (arXiv:2308.06595). arXiv.
http://arxiv.org/abs/2308.06595.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S. & Kreis, K. (2023). Align Your Latents: High-Resolution
Video Synthesis With Latent Diffusion Models (arXiv:2304.08818). arXiv. http://arxiv.org/abs/2304.08818.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C.,
Florence, P., Fu, C., Arenas, M. G., Gopalakrishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., … Zitkovich, B.
(2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. (arXiv:2307.15818). arXiv.
https://arxiv.org/abs/2307.15818.
Castaño, J., Martínez-Fernández, S., Franch, X. & Bogner, J. (2023). Exploring the Carbon Footprint of Hugging Face’s ML
Models: A Repository Mining Study. 2023 ACM/IEEE International Symposium on Empirical Software Engineering and
Measurement (ESEM), 1–12. https://doi.org/10.1109/ESEM56168.2023.10304801.
Chen, L., Chen, Z., Zhang, Y., Liu, Y., Osman, A. I., Farghali, M., Hua, J., Al-Fatesh, A., Ihara, I., Rooney, D. W. & Yap, P.-S. (2023).
“Artificial Intelligence-Based Solutions for Climate Change: A Review.” Environmental Chemistry Letters 21, no. 5: 2525–57.
https://doi.org/10.1007/s10311-023-01617-y.
Chen, L., Zaharia, M. & Zou, J. (2023). How Is ChatGPT’s Behavior Changing Over Time? (arXiv:2307.09009). arXiv.
http://arxiv.org/abs/2307.09009.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A.,
Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., … Zaremba, W. (2021). Evaluating Large
Language Models Trained on Code (arXiv:2107.03374; Version 2). arXiv. https://doi.org/10.48550/arXiv.2107.03374.
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S. & Amodei, D. (2023). Deep Reinforcement Learning From Human
Preferences (arXiv:1706.03741). arXiv. https://doi.org/10.48550/arXiv.1706.03741.

Chapter 2 Preview

88Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C. &Schulman, J. (2021). Training Verifiers to Solve Math Word Problems (arXiv:2110.14168). arXiv. http://arxiv.org/abs/2110.14168.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y. & Défossez, A. (2024). Simple and Controllable Music
Generation (arXiv:2306.05284). arXiv. https://doi.org/10.48550/arXiv.2306.05284.
Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
(arXiv:2305.14314). arXiv. http://arxiv.org/abs/2305.14314.
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang,W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., … Florence, P.
(2023). PaLM-E: An Embodied Multimodal Language Model (arXiv:2303.03378). arXiv. http://arxiv.org/abs/2303.03378.
Gandhi, K., Fränken, J.-P., Gerstenberg, T. & Goodman, N. D. (2023). Understanding Social Reasoning in Language Models
With Language Models (arXiv:2306.15448). arXiv. http://arxiv.org/abs/2306.15448.
Gemini Team: Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K.,
Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., … Vinyals, O. (2023). Gemini:
A Family of Highly Capable Multimodal Models (arXiv:2312.11805). arXiv. http://arxiv.org/abs/2312.11805.
Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D. & Misra, I. (2023). Emu Video:
Factorizing Text-to-Video Generation by Explicit Image Conditioning (arXiv:2311.10709). arXiv. http://arxiv.org/abs/2311.10709.
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N.,
Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., … Li, Z. (2023).
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models (arXiv:2308.11462).
arXiv. http://arxiv.org/abs/2308.11462.
Haque, A., Tancik, M., Efros, A. A., Holynski, A. & Kanazawa, A. (2023). Instruct-NeRF2NeRF: Editing 3D Scenes With
Instructions (arXiv:2303.12789). arXiv. http://arxiv.org/abs/2303.12789.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. & Steinhardt, J. (2021). Measuring Massive Multitask
Language Understanding (arXiv:2009.03300). arXiv. http://arxiv.org/abs/2009.03300.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D. & Steinhardt, J. (2021). Measuring Mathematical
Problem Solving With the MATH Dataset (arXiv:2103.03874). arXiv. http://arxiv.org/abs/2103.03874.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M. & Leskovec, J. (2021). Open Graph Benchmark:Datasets for Machine Learning on Graphs (arXiv:2005.00687). arXiv. https://doi.org/10.48550/arXiv.2005.00687.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X. & Zhou, D. (2024). Large Language Models Cannot
Self-Correct Reasoning Yet (arXiv:2310.01798). arXiv. http://arxiv.org/abs/2310.01798.
Huang, Q., Vora, J., Liang, P. & Leskovec, J. (2023). Benchmarking Large Language Models as AI Research Agents(arXiv:2310.03302). arXiv. http://arxiv.org/abs/2310.03302.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. & Narasimhan, K. (2023). SWE-bench: Can Language Models
Resolve Real-World GitHub Issues? (arXiv:2310.06770). arXiv. https://doi.org/10.48550/arXiv.2310.06770.
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H. & Szolovits, P. (2020). What Disease Does This Patient Have?
A Large-Scale Open Domain Question Answering Dataset From Medical Exams (arXiv:2009.13081). arXiv.
http://arxiv.org/abs/2009.13081.
Kıcıman, E., Ness, R., Sharma, A. & Tan, C. (2023). Causal Reasoning and Large Language Models: Opening a New Frontier
for Causality (arXiv:2305.00050). arXiv. http://arxiv.org/abs/2305.00050.

Chapter 2 Preview

89Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P. &Girshick, R. (2023). Segment Anything (arXiv:2304.02643). arXiv. http://arxiv.org/abs/2304.02643.
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G. & Grefenstette, E. (2018). “The NarrativeQA
Reading Comprehension Challenge.” Transactions of the Association for Computational Linguistics 6: 317–28.
https://doi.org/10.1162/tacl_a_00023.
Krizhevsky, A. (2009). Learning Multiple Layers of Features From Tiny Images. https://www.semanticscholar.org/paper/
Learning-Multiple-Layers-of-Features-from-Tiny-Krizhevsky/5d90f06bb70a0a3dced62413346235c02b1aa086.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K.,
Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q. & Petrov, S. (2019). “Natural Questions:
A Benchmark for Question Answering Research.” Transactions of the Association for Computational Linguistics 7: 452–66.
https://doi.org/10.1162/tacl_a_00276.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A. & Prakash, S.
(2023). RLAIF: Scaling Reinforcement Learning From Human Feedback With AI Feedback (arXiv:2309.00267). arXiv.
http://arxiv.org/abs/2309.00267.
Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J. S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H. B., Bellagente, M., Kang, M.,
Park, T., Leskovec, J., Zhu, J.-Y., Fei-Fei, L., Wu, J., Ermon, S. & Liang, P. (2023). Holistic Evaluation of Text-to-Image Models(arXiv:2311.04287). arXiv. https://doi.org/10.48550/arXiv.2311.04287.
Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y. & Wen, J.-R. (2023). HaluEval: A Large-Scale Hallucination Evaluation Benchmark
for Large Language Models (arXiv:2305.11747). arXiv. https://doi.org/10.48550/arXiv.2305.11747.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman,
B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., … Koreeda, Y. (2023).
Holistic Evaluation of Language Models (arXiv:2211.09110). arXiv. https://doi.org/10.48550/arXiv.2211.09110.
Lin, S., Hilton, J. & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods (arXiv:2109.07958). arXiv.
https://doi.org/10.48550/arXiv.2109.07958.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang,C., Shen, S., Zhang, T., Su, Y., Sun, H., … Tang, J. (2023). AgentBench: Evaluating LLMs as Agents (arXiv:2308.03688). arXiv.
https://doi.org/10.48550/arXiv.2308.03688.
Luccioni, A. S., Jernite, Y. & Strubell, E. (2023). Power Hungry Processing: Watts Driving the Cost of AI Deployment?
(arXiv:2311.16863). arXiv. http://arxiv.org/abs/2311.16863.
Luo, J., Paduraru, C., Voicu, O., Chervonyi, Y., Munns, S., Li, J., Qian, C., Dutta, P., Davis, J. Q., Wu, N., Yang, X., Chang, C.M., Li, T., Rose, R., Fan, M., Nakhost, H., Liu, T., Kirkman, B., Altamura, F., … Mankowitz, D. J. (2022). Controlling Commercial
Cooling Systems Using Reinforcement Learning (arXiv:2211.07357). arXiv. https://doi.org/10.48550/arXiv.2211.07357.
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y. & Potts, C. (2011). “Learning Word Vectors for Sentiment Analysis.” In D.
Lin, Y. Matsumoto & R. Mihalcea, eds., Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:
Human Language Technologies: 142–50. Association for Computational Linguistics. https://aclanthology.org/P11-1015.
Melas-Kyriazi, L., Rupprecht, C., Laina, I. & Vedaldi, A. (2023). RealFusion: 360° Reconstruction of Any Object From a Single
Image (arXiv:2302.10663). arXiv. http://arxiv.org/abs/2302.10663.
Mihaylov, T., Clark, P., Khot, T. & Sabharwal, A. (2018). “Can a Suit of Armor Conduct Electricity? A New Dataset forOpen Book Question Answering.” In E. Riloff, D. Chiang, J. Hockenmaier & J. Tsujii, eds., Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing: 2381–91. Association for Computational Linguistics.
https://doi.org/10.18653/v1/D18-1260.

Chapter 2 Preview

90Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D. & Zeng, A. (2023). Large Language
Models as General Pattern Machines (arXiv:2307.04721). arXiv. https://doi.org/10.48550/arXiv.2307.04721.
Mitchell, M., Palmarini, A. B. & Moskvichev, A. (2023). Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning
Tasks (arXiv:2311.09247). arXiv. http://arxiv.org/abs/2311.09247.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y. & Cohen-Or, D. (2022). Null-Text Inversion for Editing Real Images Using Guided
Diffusion Models (arXiv:2211.09794). arXiv. https://doi.org/10.48550/arXiv.2211.09794.
Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J. & Schölkopf, B. (2016). “Distinguishing Cause From Effect Using
Observational Data: Methods and Benchmarks.” The Journal of Machine Learning Research 17, no. 1: 1103–1204.
Nie, A., Zhang, Y., Amdekar, A., Piech, C., Hashimoto, T. & Gerstenberg, T. (2023). MoCa: Measuring Human-Language Model
Alignment on Causal and Moral Judgment Tasks (arXiv:2310.19677). arXiv. http://arxiv.org/abs/2310.19677.
Olabi, A. G., Abdelghafar, A. A., Maghrabie, H. M., Sayed, E. T., Rezk, H., Radi, M. A., Obaideen, K. & Abdelkareem, M. A.
(2023). “Application of Artificial Intelligence for Prediction, Optimization, and Control of Thermal Energy Storage Systems.”
Thermal Science and Engineering Progress, 39: 101730. https://doi.org/10.1016/j.tsep.2023.101730.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., … Zoph, B. (2024).
GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. & Finn, C. (2023). Direct Preference Optimization: Your
Language Model Is Secretly a Reward Model (arXiv:2305.18290). arXiv. http://arxiv.org/abs/2305.18290.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J. & Bowman, S. R. (2023). GPQA: A GraduateLevel Google-Proof Q&A Benchmark (arXiv:2311.12022). arXiv. http://arxiv.org/abs/2311.12022.
Rustia, D. J. A., Chiu, L.-Y., Lu, C.-Y., Wu, Y.-F., Chen, S.-K., Chung, J.-Y., Hsu, J.-C. & Lin, T.-T. (2022). “Towards Intelligent andIntegrated Pest Management Through an AIoT-Based Monitoring System.” Pest Management Science 78, no. 10: 4288–4302.
https://doi.org/10.1002/ps.7048.
Schaeffer, R., Miranda, B. & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? (arXiv:2304.15004).
arXiv. http://arxiv.org/abs/2304.15004.
Schneider, F., Kamal, O., Jin, Z. & Schölkopf, B. (2023). Moûusai: Text-to-Music Generation With Long-Context Latent Diffusion(arXiv:2301.11757). arXiv. https://doi.org/10.48550/arXiv.2301.11757.
Shams, S. R., Jahani, A., Kalantary, S., Moeinaddini, M. & Khorasani, N. (2021). “Artificial Intelligence Accuracy Assessment in NO2
Concentration Forecasting of Metropolises Air.” Scientific Reports 11, no. 1: 1805. https://doi.org/10.1038/s41598-021-81455-6.
Shi, Y., Wang, P., Ye, J., Long, M., Li, K. & Yang, X. (2024). MVDream: Multi-View Diffusion for 3D Generation (arXiv:2308.16512).
arXiv. http://arxiv.org/abs/2308.16512.
Soomro, K., Zamir, A. R. & Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild
(arXiv:1212.0402; Version 1). arXiv. http://arxiv.org/abs/1212.0402.
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.-H., Vuong, Q., Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., Finn, C. &Hausman, K. (2023). Open-World Object Manipulation Using Pre-trained Vision-Language Models (arXiv:2303.00905). arXiv.
https://doi.org/10.48550/arXiv.2303.00905.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D.,
Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., … Scialom, T. (2023). Llama 2: OpenFoundation and Fine-Tuned Chat Models (arXiv:2307.09288). arXiv. https://doi.org/10.48550/arXiv.2307.09288.

Chapter 2 Preview

91Artificial IntelligenceIndex Report 2024

Chapter 2: Technical Performance
Appendix

Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S. & Kambhampati, S. (2023). PlanBench: An Extensible Benchmark
for Evaluating Large Language Models on Planning and Reasoning About Change. Thirty-Seventh Conference on Neural
Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=YXogl4uQUO.
Voynov, O., Bobrovskikh, G., Karpyshev, P., Galochkin, S., Ardelean, A.-T., Bozhcnko, A., Karmanova, E., Kopanev, P., LabutinRymsho, Y., Rakhimov, R., Safin, A., Serpiva, V., Artemov, A., Burnaev, E., Tsetserukou, D. & Zorin, D. (2023). Multi-sensor LargeScale Dataset for Multi-view 3D Reconstruction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 21392–403. https://doi.org/10.1109/CVPR52729.2023.02049.
Walker, C. M. & Gopnik, A. (2014). “Toddlers Infer Higher-Order Relational Principles in Causal Learning.” Psychological Science25, no. 1: 161–69.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L. & Anandkumar, A. (2023). Voyager: An Open-Ended
Embodied Agent With Large Language Models (arXiv:2305.16291). arXiv. http://arxiv.org/abs/2305.16291.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. & Zhou, D. (2023). Chain-of-Thought Prompting
Elicits Reasoning in Large Language Models (arXiv:2201.11903). arXiv. https://doi.org/10.48550/arXiv.2201.11903.
Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S. & Tompson, J. (2023). Robotic Skill Acquisition
via Instruction Augmentation With Vision-Language Models (arXiv:2211.11736). arXiv. https://doi.org/10.48550/arXiv.2211.11736.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D. & Chen, X. (2023). Large Language Models as Optimizers(arXiv:2309.03409). arXiv. http://arxiv.org/abs/2309.03409.
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., Zhao, Z., Watanabe, S. & Meng, H.
(2023). UniAudio: An Audio Foundation Model Toward Universal Audio Generation (arXiv:2310.00704). arXiv.
http://arxiv.org/abs/2310.00704.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y. & Narasimhan, K. (2023). Tree of Thoughts: Deliberate Problem
Solving With Large Language Models (arXiv:2305.10601). arXiv. http://arxiv.org/abs/2305.10601.
Zellers, R., Bisk, Y., Farhadi, A. & Choi, Y. (2019). From Recognition to Cognition: Visual Commonsense Reasoning(arXiv:1811.10830). arXiv. http://arxiv.org/abs/1811.10830.
Zhang, L., Rao, A. & Agrawala, M. (2023). Adding Conditional Control to Text-to-Image Diffusion Models (arXiv:2302.05543).
arXiv. http://arxiv.org/abs/2302.05543.
Zhang, Z., Han, L., Ghosh, A., Metaxas, D. & Ren, J. (2022). SINE: SINgle Image Editing With Text-to-Image Diffusion Models(arXiv:2212.04489). arXiv. https://doi.org/10.48550/arXiv.2212.04489.

Chapter 2 Preview

92