Why are Graphics Systems so Fast?
Pat Hanrahan
Pervasive Parallelism Laboratory
Stanford University
PACT Keynote
September 14, 2009

Answer 1
Simulating virtual worlds
requires high performance

Page 1Modern Graphics Pipeline
Application
Command
Geometry
Rasterization
TextureFragment
Display

NVIDIA Historicals
Year

Product

Tri rate

CAGR

Tex rate

CAGR

1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008

Riva ZX
Riva TNT2
GeForce2 GTS
GeForce3
GeForce Ti 4600
GeForce FX
GeForce 6800 Ultra
GeForce 7800 GTX
GeForce 7900 GTX
GeForce 8800 GTX
GeForce GTX 280

3m
9m
25m
30m
60m
167m170m
940m
1400m
1800m

3.0
2.8
1.2
2.0
2.8
1.0
3.9
1.5
1.3

100m
350m
664m
800m
1200m
2000m
6800m
10300m15600m36800m48160m

3.5
1.9
1.2
1.5
1.7
2.7
2.0
1.4
2.3
1.3

1.7
Yearly Growth well above 1.5 (Moore’s Law)

Page 2

1.8

Answer 2
Cinematic games and media
drive large GPU market∴ Afford the cost of SOA designs

ATI Radeon 4870

 55 nm process 958 million transistors 1016-wide 5-op cores
 750 Mhz clock 256-bit GDDR memory
GDDR3/4/5 @ 3.6 Ghz = 115.2 GB/s
 334 Watts(R300 die)Page 3

NVIDIA GTX 280
 65 nm TSMC process 1.4 biillion transistors
 575 mm^2
 240 scalar processors
1.3 Ghz clock rate
 512-bit GDDR memory
GDDR @ 1.1 Ghz = 141.7 GB/s
 236 Watts

Answer 3
GPUs efficiently use
semiconductor technology

Page 4

Scaling Laws
Moore’s Law
 Number of transistors doubles every 18 months
 Number of transistors increases by ~50% / yr
 Feature size decreases by ~25% / yr
 Gate delay decreases with feature size – by ~25% / yr

Semiconductor capability =Number of transistors / Switching speed 50% (number) + 25% (speed)

The Capability Gap
52%/year30:174%/year

19%/year
1,000:1

Graph courtesy of Bill Dally

Page 530,000:1

Answer 4
GPUs cleverly employ
many forms of parallelism
in innovative ways

3 Axes of Parallelism

MultiCore

SIMD
VectorMultiThread

Page 6

GPU Architectures
A Closer Look at GPUs
Kayvon Fatahalian and Mike Houston
Communications of the ACM. Vol. 51, No. 10 (October 2008)
From Shader Code to a Teraflop: How Shader Cores Work
Kayvon Fatahalian
Beyond Programming Shading, SIGGRAPH 2009 Course Notes

GeForce G80 Series GPU
HostInput Assembler

Rasterization

SPSPSPSP

SP

Geometry Thread

SPSPSPSP

SP

Pixel Thread

SPSPSPSPSP

TF

TF

TF

TF

TF

TF

TF

TF

L1L1

L1

L1

L1

L1

L1

L1

L2FB

L2FB

L2FB

L2FB

Page 7

L2FB

SP

L2FB

Thread Processor

Vertex Thread

Shader Model 4.0 Architecture

Parameters

InputRegisters
Program
Textures

OutputBasic Vertex Program
DP4DP4DP4DP4DP3DP3DP3DP3DP3MOV
LITMADMADENDo[HPOS].x, c[0], v[OPOS];# Transform pos.
o[HPOS].y, c[1], v[OPOS];o[HPOS].z, c[2], v[OPOS];o[HPOS].w, c[3], v[OPOS];
R0.x, c[4], v[NRML];# Transform normal.
R0.y, c[5], v[NRML];
R0.z, c[6], v[NRML];
R1.x, c[32], R0;# R1.x = L DOT N'
R1.y, c[33], R0;# R1.y = H DOT N'
R1.w, c[38].x;# R1.w = specular
R2, R1;# Compute lighting
R3, c[35].x, R2.y, c[35].y;# diffuse + ambiento[COL0].xyz, c[36], R2.z, R3; # + specular

Page 8

G80 “core” / CUDA Architecture
Each core Thread block

SIMD (SIMT)

8 functional units

MADD + MUL

16/32 “warp”

 32 thread blocks per core
Local Storage 1024 “threads” total

Each unit 2-3 cores 16 KB shared memory

Critical Inner Loop for Graphics
ps_2_0
DCL
DCL
DCL_2D
TEX1D
MUL
MOVt0.xy
v0.xyzw
s0r0, t0, s0r1, r0, v0oC0, r1# Interpolate t0.xy# Interpolate v0.xyzw# Declaration – no code
# TEXTURE LOAD!
# Multiply# Store to framebuffer

The program must run at 100% efficiency
Short inner loop
Very little state (few registers)
Random memory (texture) access

Page 9

GPU Multi-threading
Change thread after texture fetch/stall
frag1

frag2

frag3

frag4

Run until stall attexture fetch(multiple instructions)

NVIDIA GeForce GTX 285 “core”…
= SIMD functional unit, control
shared across 8 units= multiply-add= multiply= instruction stream decode= execution context storage

Page 10Typical Chip16 cores
8 mul-add ALUs per core(128 total)= 256 GFLOPs (@ 1GHz)“Enthusiast” Chip!

32 cores x 8 SIMD functional units x 3 flops/cycle x 1.3 Ghz = 933 Gflops

Page 11

NVIDIA GeForce GTX 285• NVIDIA-speak:– 240 stream processors– “SIMT execution”• Generic speak:– 30 cores– 8 SIMD functional units per core

AMD Radeon HD 4890 “core”…

= SIMD VLIW functional unit,
control shared across 16 units= instruction stream decode= execution context storage= multiply-add

Page 12

AMD Radeon HD 4890 “core”…

• Groups of 64 [fragments/vertices/etc.] share instruction stream(AMD doesn’t have a fancy name like “WARP”)– One fragment processed by each of the 16 SIMD units– Repeat for four clocks

AMD Radeon HD 4890
Tex

Tex
…

…

Tex

Tex
…

…

Tex

Tex

…

…

Tex

Tex

…

…

Tex

Tex
…

…

26

Page 13

AMD Radeon HD 4890
• AMD-speak:– 800 stream processors– HW-managed instruction stream sharing (like “SIMT”)• Generic speak:– 10 cores– 16 SIMD functional units per core
– 5 ALUs per VLIW unit per SIMD lane27

Larrabee

Larrabee: A many-core x86 architecture for visual computing,D. Carmean, E. Sprangle, T. Forsythe, M. Abrash, L. Seiler, A. Lake, P.
Dubey, S. Junkins, J. Sugerman, P. Hanrahan, SIGGRAPH 2008
(IEEE Micro 2009, Top Pick)

Page 14

Larrabee Core
Thread Dispatch

L1 Instruction Cache

ALUAGU

ALU
AGU

L1 Data Cache

Vector processor interface

Instruction Decode

Separate scalar and vector units
Separate register files
In-order IA scalar core
Vector unit: 16 32-bit ops/clock
Short execution pipelines
Fast access from L1 cache
Direct connection to L2 cache
Prefetch to manage L1/L2 caches

256K L2 Cache

RingVector Processing Unit
Mask Registers
StoreConvert
BufferVector
L1Mask Registers
Reg File
Data Cache
NumericConvert
NumericConvert
Convert• Vector instructions support– Fast, wide read from L1 cache– Numeric type conversion and data
– Rearrange the lanes on register read
– Fused multiply add (three arguments)– Int32, Float32 and Float64 data

• Augmented vector instruction set
16-wide16-wide Vector
Vector ALU
ALU– Scatter/gather for vector load/store– Mask registers select lanes

L1 Data Cache
Vector Mask Registers

Page 15Example LRBni Vector Instructions
1.
2.
3.
4.
5.
6.

multiply-add:

vmadd132ps v1, v2, v3

Address computation
LoadUpconvert
MultiplyAddMask

mask the writing of the elements:

vmadd132ps v1 {k1}, v2, v3
source from memory

vmadd132ps v1 {k1}, v2, [rbx+rcx*4]
memory source undergoes format conversion

vmadd132ps v1 {k1}, v2, [rbx+rcx*4]{float16}

Diﬀerent Notions of “SIMD”• Option 1: Scalar and vector instructions– Small number of threads– Intel/AMD x86 SSE, Intel Larrabee• Option 2: Scalar instructions ⇒ implicit vectors– Only scalar instructions; hardware merges instruction streams– Each instruction is executed a small number of times– NVIDIA GeForce (“SIMT” warps), AMD Radeon architectures

Page 16

Various Notions of Threads
Cores: Each runs multiple threads
Thread: HW-managed context (hide short unpredictable latencies)…
More Threads (up to 4 per core, share memory via L1 & L2 caches)

Larrabee: Core, Threads

Various Notions of Threads
Cores: Each runs multiple threads
Thread: HW-managed context (hide short unpredictable latencies)…

StrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandStrandFiber: SW-managed context (hides long predictable latencies)

16-wide vector unit

16-wide vector unit…
More Fibers running (typically 2-10, depending on latency to cover)…
More Threads (up to 4 per core, share memory via L1 & L2 caches)

Larrabee: Core, Threads, Fibers, Strands

Page 17

Intel Larrabee “core”

Each HW context:
32 vector registers
32 KB of L1 cache
256 KB of L2 cache= SIMD vector unit,
control shared across 16 lanes
additional scalar unit
= mul-add= instruction stream decode= execution context storage/
HW registersIntel Larrabee• Intel speak:– We won’t say anything about core count or clock rate
– Explicit 16-wide vector ISA– Each core interleaves four x86 instruction streams• Generic speak:– That was the generic speak

Page 18

Intel Larrabee

…

?

…

…
?
…
Tex

Tex

…?…

Page 19Tex

Tex

Larrabee
Each Larrabee core is a complete IA core Context switching & pre-emptive multi-tasking
 Virtual memory and page swapping Fully coherent caches at all levels of the hierarchy

Efficient inter-block communication
 Ring bus for full inter-processor communication
 Low latency high bandwidth L1 and L2 caches
 Fast synchronization between cores and caches

RecapPage 20

CPU-“style” Cores

Fetch/
DecodeOut-of-order control logic
Fancy branch predictor

ALU(Execute)

Memory pre-fetcher

Execution
Context

Data cache(A big one)GPU-style Cores
Fetch/
Decode
ALU 1

ALU 2

ALU 3

ALU 4

ALU 5ALU 6

ALU 7

ALU 8

Ctx

Ctx

Ctx

Ctx

Ctx

Ctx

Ctx

Ctx

Shared Local Data

Page 21

Three Key Ideas
1.

Simplify the core.
 Remove high-overhead logic to control out of order

execution, branch predication, etc.

2.

Exploit the efficiency of SIMD processing Share instructions and replicate functional units

3.

Use many threads to hide memory latency
 Smaller caches, but still need thread state If you have enough thread state, never a stall

Optimizing for Throughput
Hypothetical Core design experiment:
Specify a throughput-optimized processor with same
area and power of a standard dual core CPU

20 times greater throughput for same area and power

Page 22

iPhone 3GS

Page 23

Apple/Samsung SoC (CPU, GPU, Mem)

Samsung ARM Cortex A8

Page 24

Imagination PowerVR SGX535

SGX520 3.5M Tri/S, 125M Pix/s @ 100 Mhz

Heterogenous “Fusion” Architectures
Emergence of a hybrid processor
 2-8 CPUs 16-64 GPUs
 Hardware for video compression/decompression
 …

Plans announced by AMD and Intel

Page 25

Answer 5Graphics Systems are Programmed
at a High-Level of Abstraction(Utilize Domain-Specific Languages)

Brook
Ian Buck
PhD Thesis
Stanford University
Brook for GPUs: Stream computing on graphics hardware,
I. Buck, T. Foley, D. Horn, J. Sugarman, K. Fatahalian, M. Houston,
P. Hanrahan, SIGGRAPH 2004
CUDA: Scalable parallel programming made clear,
J. Nickolls, I. Buck, K. Skadron, and M. Garland,
ACM Queue, April 2008

Page 26

Brook Example
kernel void foo (float a<>, float b<>,out float result<> ){result = a + b;}
float a<100>;
float b<100>;
float c<100>;foo(a,b,c);for (i=0; i<100; i++)c[i] = a[i]+b[i];Current Statistics: September 13, 2009
Client type

Current
TFLOPS*215

Active Processors

Mac OS X/Intel22

5,063

Linux77

45,028ATI

1,027

10,069NVIDIA

1,992

16,736

PS/3

1,075

38,110

Total

4,412

347,825

Windows*TFLOPs is actual folding flops, not peak values

Page 27

225,721

Domain-Specific Languages
Graphics (GRAMPS)
Molecular dynamics (GROMACS)
Physical simulation on meshes (Liszt)
Data-parallel programming (Kore)
Statistics/machine learning and data analysis (Bern)
Computer vision and imaging
Brain simulationAutonomous vehicles…

Wrap-Up

Page 28

Questions and Answers
Why are graphics systems so fast?
1. Simulating virtual worlds requires high performance
2. Cinematic games and media drive large GPU market
3. GPUs (more) efficiently use semiconductor resources
4. GPUs employ many forms of parallelism in innovative
ways (core, thread, vector)
5. GPUs are programmed at a high-level
Why are other compuer systems so slow / inefficient?

Architectural Issues
High-throughput processor design
 SIMD vs. blocked threads (SIMT) Software- vs. hardware-managed threads

Processor of the future likely to be a hybrid CPU/GPU Why? Heterogeneous workload Small number of traditional CPU cores runninga moderate number of sequential tasks

 Large number of high-throughput GPU cores running

data-parallel work Special hardware for tasks that need to be power

efficientPage 29

Opportunities
Current hardware not optimal
 Incredible opportunity for architectural innovation

Current software environment immature Incredible opportunity for reinventing parallel

computing software, programming environments
and language

Acknowledgements
Bill Dally

Ian BuckAlex Aiken

Kayvon Fatahalian

Eric Darve

Tim Foley

Vijay PandeDaniel Horn

Bill MarkMichael Houston

John Owens

Jeremy Sugerman

Kurt Akeley

Doug CarmeanMark Horowitz

Michael Abrash

Funding: DARPA, DOE, ATI, IBM, INTEL, NVIDIA, SONY

Page 30

Questions?

Page 31