1Wから1MWまで同一アーキテクチャでスケールするAIアクセラレータ

Tenstorrent Japan S.FAE Yasuhiro Ito
Oct 2023

Topics• About Tenstorrent & AI-chips + RISC-V CPU• Scalable architecture and Software development kits

• Single Accelerator & SDK cover from Edge to Cloud .

2

Our company & productsAbout Tenstorrent• Based in North America, with offices in Tokyo, Toronto,
Santa Clara, Austin, Belgrade, and Bengaluru.
• Tenstorrent builds the most innovative AI products:• Inference and Training, CNNs, LLMs, and NLPs
• Powerful software stacks for models & bare metalprogramming• Tenstorrent created the highest performing RISC-V CPU
technology in the world• Led by industry veteran hardware engineer andCPU architect, CEO Jim Keller.

4

Tenstorrent CPU TeamJim Keller

Wei-Han Lien

Jim Montanaro

CEO, Digital Alpha
processor, Apple A
series, AMD Zen,
Tesla Autonomous
Driving system

Chief CPU Architect:
Apple, PA Semi, AMD

PD Fellow:
Apple, AMD

Dan Bailey

Srikanth Arekapudi

Yasuo Ishii

Senior Fellow:
Tesla, AMD,DEC

RTL/DV Fellow:
Cerebras, AMD

Architecture
Fellow: Arm, NEC5

Current products
We deliver AI/ML to customers
Grayskull
For Infenrence

GrayskullWormhole

Wormhole
For Inference & Training

IPWorkstations
PCIe Cards

ServersGalaxy

AI Cloud6

PCIe Cards

n300

n150

e300e150

e75

TechnologyWormhole, dualchip

Wormhole, single
chip

Grayskull, dual chip

Grayskull, single
chip

Grayskull, single
chip

Form Factor¾ length¾ length¾ length¾ length½ length

Performance

Silicon TOPS 540

Silicon TOPS 315

Silicon TOPS 442

Silicon TOPS 276

Silicon TOPS 220

DRAM

24 GB, 1152
GB/sec12 GB, 576
GB/sec16 GB, 200 GB/sec

8 GB, 100 GB/sec

8 GB, 100 GB/sec

Network

PCIe 4, 16x 100GbPCIe 4, 16x100Gb

PCIe Gen 4

PCIe Gen 4

PCIe Gen 4

Power300W max

150W max

300W max

150W max

75W max

7

Wormhole Products (2nd Gen device for AI at scale)
12nm AI Accelerator on PCIe Gen 4

N300s/d (Nebula, single or dual chip config available)• Modular device with 1.6TB onboard ethernet• Natively scalable to an arbitrary number of devices• High performance at low costGalaxy••••

High-density AI servers in 4U enclosures
Comprised of 32 x n300s devices
Includes backplane interconnect, active cooling
7PFLOP (BF8) at 7.5KW

8Tenstorrent wins on Perf/$ (and Perf/W)
Galaxy wins on OOB Large Language Model Performance and Price

Out of the Box Performance

1.7x MORE

nVidia DGX
Tenstorrent Galaxy

Performance
Deliverson BERT-large model

PricenVidia DGX

vs.
nVidiaDGX

Tenstorrent
Galaxy8x A100$120,000

32x Wormhole$80,000(est. ~6.5 kW)(est. ~6.5 kW)

Tenstorrent Galaxy

33%

Lower Cost$$$$$

Delivers

Price Performance

2.6X MORE

nVidia DGX
Tenstorrent Galaxy

on BERT-large model* List price comparison

Price Performance
Deliverson BERT-large model* List price comparison
9Tensix: Scalable AI accelerator

AI Chip Roadmap
HeterogenousChiplet
2021

2022

2023

Grayskull

Wormhole

Black HoleQuasar

Grendel

ML Processor

Networked MLProcessor

Standalone MLComputer

Low Power, Low
Cost ML Chiplet

Highly Configurable andPerformant ML Chiplet• 12nm, 276 TFLOP (FP8)• 12nm, 328 TFLOP (FP8)• 200 GB/S Scale-out Ethernet• 6nm, 600mm2• Small RISC-V cluster• Heterogenous compute2024• ML Chiplet• CPU + ML chiplets

Multiple Generations of AI Accelerators
All Powered by the Scalable High PerformanceTensix AI CoreTenstorrent Confidential

1Tenstorrent ML Accelerator Architecture• RISC-V CPUs in a tile support
flexible data manipulation

Grayskull: 120 Tensix cores• Example• One for NoC receive data
• One for NoC transmit data
• Three for Compute engine
12

Mapping Operation to Tensix• Processing elements are assigned to each graph node
• Software balances the amount of assigned core per graph node
• Just like in CPU pipelines, the data moves through the pipe

13

GPU vs. Tenstorrent Memory Bandwidth

VS.

GPU Memory

Tenstorrent Memory

A GPU needs to reload op
parameters every time itswitches to the next op

In a Tenstorrent chip, the outputs of the ops feed the next
op over the NOC, and only the first and the last op on thechip communicate with DRAM.

14

‘Scale-out’ Microarchitecture• Large AI models can bemapped to multiple chips• Data flows can go
through chip to chip

• Scale-out multiple racks

15

SDK

Tenstorrent Software – Two Distinct Approaches

METAL: Open Accessto Tenstorrent
Hardware & Software

Graphyte: Run any
model right away

Graphyte

Great for production
customers who wantto get models up andrunning with ease.
They want flexibility,
but don’t have time toprogram newoperations or contact
Nvidia every timesomething changes.

METAL

Great for development
customers who want tocustomize their models,
write new ones or evenrun non-machinelearning code. No blackboxes, encrypted APIsor hidden functions.

17ML compiler stack (high level)•Fully automated path from all
popular ML framework to
optimized implementation•High quality results with nomanual effort•

Same compiler targets one chipor many thousands of chips

Hardwareagnostic
parallelsolution

Hardwarespecificbackend18

We support Temporal and Spatial Execution
Epoch1
op3
op2
op1
op0

WH0

WH
1

op0

op2

op1

WH0

WH
2

Temporal: Ops run one after the other, eachusing all available resources
Every op runs as fast as it canWH
1

PROS:

••
•
•

High DRAM overhead in r/w intermediate
data
Reconfigure penalty between ops
Small ops do not use available hardware
efficiently•••

Local memory / cache reuse for model
parameters
Intermediate data travels on NOC
Zero reconfiguration and intermediate
data storage overhead
Very high utilization of resourcesCONS:

••

WH0

WH
2

Spatial: Each op is assigned fixed resources,
according to their size. Model runs as apipeline.

CONS:

op3

op1

PROS:

•Epoch0op0

op3

op
2Large models could require large numberof devices to fit
Pipeline overhead for fill/drainWH1

Temporal/Spatial Mix: Model is divided into
“epochs”, where each epoch spatially fits
within available hardware. Epochs run
sequentially. Small ops are fused into larger
ones.
PROS:

•••

Fits on available hardware
High utilization within epochs due tospatial computationReconfigure & intermediate storage
overhead limited due to a smaller of
number of transitions

19Running Huggingface Inference in Pytorch vs. Graphyte

Running HF with PyTorch

Running HF withGraphyte

Over 40 standard models running

NLP

Computer Vision

BERT

ResNetResNeXt

Wav2Vec

YOLOv5

VideoPose

UniSpeech

ViT

VGG

ViLT

ALBERT

DenseNet

HRNet

NBeatsRoBERTa

U-Net

MNIST

DeepFM

DistilBERT

MobileNetV3

DeepCoNN

GPT Neo

MobileNetV2DALLE VAE

MobileNetV1

ConvNeXt

EfficientNetV2

GhostNet

XLM

EfficientNet

FCN

SqueezeBERT

VoVNetOpenPoseGPT-2
BART
T5

GPT-J
OPTXGLM

Other

With another 50 in thepipeline…

21

METAL Delivers More Control and Capabilities than CUDA

METAL Advantages
over CUDA:
1.

Kernels are pure C++
with APIs

2.

De-coupled datamovement & compute –
optimize compute anddata movement
separately

3.

Different Cores can run
different kernels withpipelines connecting them4.

Direct Control of SRAM
and DRAM

METAL
Deep
Learning OPs

CUDA/OpenCL
DeepLearning OPs

Collective
Comms OPs

VS.

Collective
Comms OPs

Metal ”Read->Compute->Write” kernels running on a Tensix Core
Data movementkernel

RISC-V1
Data into the coreComputekernelRISC-V2

RISC-V
3

Data movement
kernelRISC-V
4

Matrix & Vector
engines

SRAMRISC-V
5
Data from the core

AI HW scalability : Datacenter ~ Edge

Auto/Electric industries

29Tenstorrent’s Offerings
ProductIP

Chiplet

Chip

Card

Systems

Cloud

RISC-VAI Acc30

Tenstorrent Example Vertical: Automotive
Tenstorrent AI andRISC-V IP deliver
the compute
power that ADAS
and IVI require

Chiplet approach
reduces cost while
accelerating design
and production
schedules.

Automotive
companies canown their own
silicon workingwith Tenstorrent

Power Consumption
is critical: Tenstorrent
technology scales
from MW to mW

Summary• Scaleable Tensix cluster + reference server appliance• SDK handles complexity and parallelis• Edge deployment, Training in datacenter with same Framework, Architecture32

Thank You!