SWEST / ACRi 共同企画セッション

AIを加速するAI Engineのアーキテクチャ解説とプログラミング体験
ACRiルーム副室長 / ザイリンクス株式会社（AMD-Xilinx）
安藤 潤©︎ 2022 Adaptive Computing Research Initiative - ACRi

1

アウトライン• ACRiルームアカウント登録のお願い• 自己紹介• AI Engineアーキテクチャ解説• Versal Architecture AI Engine Technical Module• ハンズオン©︎ 2022 Adaptive Computing Research Initiative - ACRi

2

ACRiルームアカウント登録のお願い©︎ 2022 Adaptive Computing Research Initiative - ACRi

3

ACRiとは• アダプティブコンピューティング研究推進体• https://www.acri.c.titech.ac.jp/• FPGAを盛り上げる会• 5大学、27社が参加（2022/7現在）• 主な活動• ウェビナー、討論会、ハンズオン開催• https://acri.connpass.com/• https://www.youtube.com/channel/UCL15_5A9JKcVrmFUcMto6-Q• ACRiブログ• https://www.acri.c.titech.ac.jp/wordpress/• ACRiルーム（無償FPGA利用環境）• https://gw.acri.c.titech.ac.jp/wp/©︎ 2022 Adaptive Computing Research Initiative - ACRi

4

ACRiルーム• アカウント登録のお願い• ハンズオンではACRiルームが提供するサーバーを利用します• ハンズオンに参加される場合は、8/26までにアカウントを申請し、ご使用になるPCからリモートデスクトップ接続できることを事前に確認してください（ハンズオン当日のサーバーの予約は不要です）• アカウント申請方法• https://gw.acri.c.titech.ac.jp/wp/manual/apply-for-account• サーバーの利用方法• https://gw.acri.c.titech.ac.jp/wp/manual/how-to-reserve• Windows 10からPower Shellを使う場合• https://www.acri.c.titech.ac.jp/wordpress/archives/12855©︎ 2022 Adaptive Computing Research Initiative - ACRi

5

自己紹介

©︎ 2022 Adaptive Computing Research Initiative - ACRi

6

自己紹介
• 組み込みと私• 液晶ディスプレイコントローラ向けのデジタルIP開発• アルゴ、ソフト、RTL、FPGA、ES• LPプロセス、案件たくさん• 8051ファームウェア• I2C、SPI• ザイリンクスに組み込みエンジニア（候補）として入社⇒ 現在はデータセンター向け製品担当©︎ 2022 Adaptive Computing Research Initiative - ACRi

7

AI Engineアーキテクチャ解説©︎ 2022 Adaptive Computing Research Initiative - ACRi

8ハンズオン©︎ 2022 Adaptive Computing Research Initiative - ACRi

9ハンズオンの流れ
1. ACRiルームのサーバーにリモートデスクトップ接続•

as101~as105をご利用ください（予約なしで利用できます）

2. チュートリアルをコピーして起動$ cd /scratch/$USER$ cp -a /tools/repo/acri-room/aie-tutorial .
$ cd aie-tutorial$ ./start.sh3. ハードウェアエミュレーション（hw_emu）を選択して進めてください（実機で動作させる場合は後日as005でお試しください）©︎ 2022 Adaptive Computing Research Initiative - ACRi

10©︎ 2022 Adaptive Computing Research Initiative - ACRi

11

[Public]

Versal® Architecture AI Engine
Technical Module
Florent Werbrouck
AI Engine & DSP Technical Marketing Engineer© Copyright 2022 AMD

[Public]

AgendaIntroduction and Motivation
Architecture Overview
AI Engine (AIE) vs AI Engine-Machine Learning (AIE-ML)
Software and Tools
System Partitioning for Versal® ACAP
AI Engine Power & XPE

Summary2© Copyright 2022 AMD

[Public]

Introduction and Motivation3© Copyright 2022 AMD

[Public]

Motivation for AI Engine
AI IS EVERYWHERE
5G Wireless
Radio

Technology Not
Scaling

Data Center
Workloads

Smart
Factories

Moore’s Law / Dennard Scaling
Performance & Power Scaling
Traditional Single / Multi-core

ADAS & AD

Compute Density
Real Time Capability
Power Efficiency

Dynamic Markets Require Adaptable Compute Acceleration
4© Copyright 2022 AMD

MachineLearning

Applications More
Demanding[Public]

Conceptual Device with High Compute Applications need (1) cost reduction, (2) power reduction, (3) more compute, (4) more programmability
 How does AMD-Xilinx meet the demands of these evolving applications?

16nm Generation(Zynq® UltraScale+™ MPSoC)

7nm Generation

IO

AI Engine Array

GT

PL

PL

PL

PL

GT
GT

GT

IO

GT

PL

PL

Processing
System
& PMC

PL

PL

PL
PL

PL
PL

PL

PL

PL

PL

PL

GT

GT

GT

PL

PL

PL

PL

PL

PL

PL

PL

Processing
System& PMC

GT

Goal: Increase Compute Density and Silicon Efficiency
5© Copyright 2022 AMD

IO
IO

GT

IO

GT

PL

PL

IO
GT
GT

PL

PL

PL

GT

[Public]

Versal, Industry’s First ACAP

 Heterogeneous Acceleration
 For Any Application

 For Any Developer

6© Copyright 2022 AMD

[Public]

Delivering Adaptable Compute Acceleration
CPU

Single  Multi-CoreGPU(Parallel)

ACAP

ASIC
AI
EnginesCache

SW Programmable

✓

✓

✓

✓

HW Adaptable

–

–

✓

–

Workload Flexibility

✓

✓

✓

–

High Throughput and Low Latency

–

–

✓

✓

–

–

✓

✓

Device / Power Efficiency
Development Time& Complexity

MonthsWeeks
7© Copyright 2022 AMDYears

[Public]

Introducing the AI Engine
Target Applications

EngineAI
EngineAI
EngineMemoryEngineAI
EngineAI

MEMORYAIAI
EngineMEMORYEngineMemoryEfficientAI
EngineAI

MEMORYMEMORY
MemoryDeterministic

MemorySW Programmable 5G Wireless – Radio, Baseband
 Data Center - AI Inference ISM – Computer Vision, Genomics
 ADAS/AD – Embedded Vision
 A&D – Radar, Sonar

SIGNAL PROCESSINGARTIFICAL INTELLIGENCE

CNN

LSTM

WIRELESSCOMPUTER VISION

MLP

Adaptable. Intelligent.
8© Copyright 2022 AMD

[Public]

AI Engine
Hardened Compute, Memory, and Interconnect
Huge performance improvements vs. UltraScale+™ FPGAs
 Up to 8x compute density @ 40% lower power

1GHz+ VLIW / SIMD vector processors
 Versatile core for ML and other advanced DSP workloads

Massive array of interconnected cores Instantiate multiple tiles (10s to 100s) for scalable compute

Terabytes/sec of interface bandwidth to other engines
MEMORYSW programmable for any developer

AI
AI
EngineEngineMemory Implement core application with AI for “Whole App Acceleration”

MEMORY
Memory Direct, massive throughput to adaptable HW engines
AIAI
EngineEngineEngine9© Copyright 2022 AMD

AI
AIEngineEngineMEMORYAIAI
EngineMemory Library-based design for ML framework developers

MEMORY
Memory C programmable, compile in minutes

[Public]

AI Engine: AMD-Xilinx Reinvents Multi-Core Compute
Traditional Multi-core

AI Engine Array(cache-based architecture)(intelligent engine)

Block 0

• Blocking limits
compute• Timing notdeterministic

D0

AI
Engine

L1

L2
DRAM

AI
EngineAI
EngineAI
EngineMemoryAI
EngineAI
EngineMemoryL1• Non-blocking• Deterministic

MemoryFixed, sharedInterconnect

D0

L0

AI
EngineMemoryL0AI
EngineMemoryL0

Dedicated
Interconnect

Memory

L0

D0

core

MemoryL0

core core

MemoryL0

core

Memorycore core

Block 1AI
EngineLocal, Distributed Memory• No cache misses• Higher bandwidth• Less capacity required

Data Replicated• Robs bandwidth• Reduces capacity
10© Copyright 2022 AMD

[Public]

Software Programmable: Any Developer
1 Design

3 Run
Programming
Abstraction Levels

Frameworks

C/C++
C/C++

4G/5G/Radar
LibraryAI
Library

Architecture
Overlay
VisionLibrary

Data Flow
w/ Xilinx libraries

Kernel Program
Data Flow w/ user
defined libraries

2 Compile
AI Engine Compiler

11© Copyright 2022 AMD

[Public]

Hardware Adaptable: Accelerating the Whole Application
Scalar, Sequential Flexible Parallel Compute,& Complex Compute
Data manipulation
Adaptable

ScalarML & Signal Processing
Vector, Compute IntensiveIntelligent
AI EnginesArm
Dual-Core
Cortex-A72

Heterogeneous Accelerationfrom Data Center to the Edge

128 GB/s ofMemory B/W
per CoreVideo + AI
Genomics + AI
Risk Modeling + AI

ArmDual-Core
Cortex-R5F

Database + AI
Network IPS + AI

PROGRAMMABLE NETWORK ON CHIP

I/O
Any-to-Any
Connectivity

Storage + AI

Custom Memory
Hierarchy

TB/s of Bandwidth
PL-to-AI Engine

Delivering Deterministic Performance & Low Latency
12© Copyright 2022 AMD

[Public]

AI Engine Delivers Scalable Compute & Faster Compile
AI Engine Compile
In Minutes

A160

 343 AI Engine design (86% of VC1902)

140

2,880 min

 Ultrascale+™ P&R takes many hours

120

INT8 TOPs

100
80

Example CNN Use-Case AI Engine compiles in ~3 minutes

2,400 min

7.5x

4.5x

AI Engines(133 TOPs)(Compile time for PL shell design not included)

60

40

3 min
20

DSP Engines
0

VU9P

VU13P

VC1902(3 SLR)(4 SLR)(Mono)Long Compile Time
(Hours to Place and Route)13© Copyright 2022 AMD

343 AI Engines, 86% of VC1902

[Public]

AI Engine Application Performance & Power Efficiency
Image Classification

Massive MIMO Radio(GoogleNet, <1ms)(DUC, DDC, CFR, DPD)
Versal® ACAP with AI Engine
UltraScale+™ Device

20x

5x

40%
Less
Power

ML Inference

5G Wireless14© Copyright 2022 AMDPower[Public]

Architecture Overview

15© Copyright 2022 AMD

[Public]

Multiple Versions Optimized for Specific Applications
BEAMFORMING,
RADAR PROCESSING,
HIGH PERF. COMPUTING

CNN, RNN, MLP

 Advanced Signal Processing

Machine Learning 

AIE-MLAIE

AIE: Version Optimized signal processing AND ML
AIE-ML: Optimized for ML inference applications
In following slides, we refer to AI Engine as the technology (common to AIE and

AIE-ML) and to AIE/AIE-ML as the versions of this technology

16© Copyright 2022 AMD

[Public]

AI Engine: Terminology
Versal® ACAP

AI Engine Array

PMC

AI
EngineAI
EngineAI
EngineMemoryArm
Cortex-R5

MemoryMemoryAdaptable
HardwareAI
EngineMemoryAI Engines

Arm
Cortex-A72

AI Engine Tile
AI Engine
ScalarRegister
File

Scalar ALU

VectorRegister
File

Non-linear
Functions

Scalar Unit

Fixed-Point
Vector Unit

Interconnect

Floating-Point
Vector Unit

ISA-based
Vector Processor
AI Vector
Extensions

Vector Unit

AGU

AGU

AGU

Load Unit A

Load Unit B

Store Unit

Memory Interface

Instruction Fetch& Decode Unit

5G Vector
Extensions

Stream Interface

17© Copyright 2022 AMD

Local
MemoryData
Mover[Public]

AI Engine: Array Architecture
Array of AI Engines Tiles• Increase in compute, memory andcommunication bandwidth
I/O

PS

AI

EngineMemoryAI

EngineMemoryAI

EngineMemoryPLAIEngineMemory• Versal AI Core VC1902 device

AI

EngineMemory• More tiles = more compute• Up to 400 per device

MemoryModular and scalable architecture
AI

EngineAI

EngineDeterministic Performance & Low Latency
18© Copyright 2022 AMD

AIEngineMemoryAI

EngineMemory• Maximize memory bandwidth

MemoryDistributed memory hierarchy

[Public]

AI Engine: Tile-Based Architecture
Non-Blocking AXI-Interconnect

I/O

PS

Memory Interface
Stream Interface

Cascade Interface

Up to 200+ GB/s bandwidth per tile
Communicate between AI Engine Cores
Access to PL and memory in the PL

PLInterconnect

ISA-based
Vector Processor
Software Programmable(e.g., C/C++)

ISA-based
Vector Processor

Local Data
MemoryProgram Memory
Application SpecificVector Extensions
E.g. AI & 5G

Cascade Interface
Partial results to next core19
© Copyright 2022 AMD

Local Memory
Multi-bank implementation
Shared across neighbor cores[Public]

AI Engine Tile
Memory Access
AXI Stream
AXI MM

AI Engine Array AI Engine core
512b SIMD vector units32b scalar RISC processor

256-bit load/read (x2) and store/write units with individual AGUs

 Up to 256KB direct core memory access

32KB local in AI Engine / 64KB local in AIE-ML

32KB north, south, east or west in AI Engine /64KB north, south and west in AIE-ML

 Streaming interconnects


AXI Memory Mapped (AXI-MM) switch


AXIS East

Program
Memory(16KB)

Instruction
Fetch &
Decode
Unit

Load & Store
Address
Generation
Units

32b Scalar
RISC Unit

Fixed Point
512b SIMD
Vector Unit

Floating Point
512b SIMD
Vector Unit

ScalarRegister Files

Configuration, control and debug

AXI-Stream crossbar switch


AXIS West

Stall
Handler

Routing N/S/E & W around the array Debug/Trace/Profile functionality
20© Copyright 2022 AMD

S2MM
DMA

MEM
I/F

Data
MemoryVector Register Files

Control,
Debug& TraceAccumulator
Stream FIFOMEM I/F

MM2S
DMA

MEM I/F

16KB program memory

MEM I/F

AXIS North

Both fixed and floating point

AXIM Switch



AXIS South

Cascade Stream

[Public]

AI Engine: Highly Parallel Processor Core
ScalarRegister
File

Scalar ALU

VectorRegister
File

Non-linear
Functions

Scalar Unit

Fixed-Point
Vector Unit
Floating-Point
Vector Unit

Vector Unit

AGU

AGU

AGU

Load Unit A

Load Unit B

Store Unit

Memory Interface

Instruction Fetch& Decode Unit

StreamInterface

Instruction Parallelism: VLIW

Data Parallelism: SIMD

7 operations / clock cycle

Multiple vector lanes› 2 Vector Loads / 1 Mult / 1 Store› 2 Scalar Ops / Stream AccessHighlyParallel› 8 / 16 / 32-bit & SPFP operands (AIE)› 4 / 8 / 16 / 32-bit & BFloat16 operands (AIE-ML)

Up to 512 MACs / Clock Cycle per Core (INT 4 x INT 8) in AIE-ML
21© Copyright 2022 AMD

[Public]

Data Movement Architecture
Streaming Communication

Memory Communication
Dataflow
Pipeline

MemoryAI
EngineMem

Dataflow
Graph

B0

MemoryMemoryB1AI
EngineAI
Engine

Mem

B2
B3

AI
EngineMem

AI
Engine

Mem

Mem

AI
EngineMem

AI
EngineStreaming
MulticastAI
EngineCascadeStreaming

Stream Interface
Cascade Interface© Copyright 2022 AMD

AIEngineAI
EngineAI
EngineMemory Interface
22

AI
EngineAI
EngineAI
EngineAI
EngineAI
Engine

NonNeighbor

MemoryAI
EngineAI
Engine[Public]

Tile Interconnect: AXIS
Single Tile Interconnect

AI Engine Array

AXI stream switches

MemoryMemoryMemoryEngineAI

EngineAI

EngineMemoryAI

EngineMemoryMemoryAI

EngineAI

Each 32b AXIS bus 5 GB/s @ fclk = 1.25 GHz
AXI Stream Master
Ports

AXI Stream Slave Ports

NumberBW GB/s

NumberBW GB/s

4 to East20

4 from East

20

4 to West20

4 from West20

6 to North30

4 from North20

4 to South20

6 from South30

Sub-total

90

Total

180

90

Slowest speedgrade @ 1 GHz ➔ 144 GB/s total bandwidth per AI Engine tile
Note: B/W is shared with adjacent AI Engine tiles; Traffic between non-adjacent tiles affects the available B/W
23© Copyright 2022 AMD

AXIS SouthAI

EngineEngineMemoryAI

EngineAI

AXIM Switch

MemoryEngineMemoryAI

AXIS North

AXIS West

AXIS East[Public]

AI Engine Integration with Versal® ACAP
I/OPS

AI ENGINE ARRAYPL TB/s of Interface Bandwidth

AI Engine to programmable logic
 AI Engine to programmable network on chip

SwitchSwitchSwitchAsync CDC

 Leveraging NoC connectivityPS manages Config / Debug / Trace
 AI Engine to DRAM (no PL req’d)

PS /
PMC

AI Engine
Interface Tiles

DMA

AXI-MM

SwitchSwitchSwitchAXI-S

Programmable
Logic24© Copyright 2022 AMD

PL
Function

NOC

Ext.
DRAM

[Public]

AI Engine to PL Interface
 AXI4-Stream switches in the PL interface

tile directly communicate with the PLHandle most of the data movement to/from the AI
Engine arrayConfigurable bit widths (32b/64b/128b)

Communication
PL → AIE interface array(Some columns are not available)#AXI Stream per
ColumnBandwidth per
Column→ North8

32 GB/s~1.3TB/s→ South6

24 GB/s~1TB/s#AXI Stream per
Interconnect

Bandwidth per
Interconnect→ North6

24 GB/s

1.2 TB/s→ South4

16 GB/s

800 GB/s

Direction

Direction
Within AI Engine Grid
(All columns are available)

Bandwidth on
VC1902

Bandwidth on
VC1902

Note: B/w calculation - 1 GHz AI Engine clock @ -1L speedgrade (0.7V), higher bandwidth is available with faster speed grade25© Copyright 2022 AMD

[Public]

AI Engine to Programmable NoC Interface
AI Engine Tile

AI Engine Tile

 In addition to the AXI4-Stream, the NoC Interface

tile provides a connection to the HNoC (horizontal
NoC)
4 streams from AI Engine to NoC



4 streams NoC to AI Engine

MM AXI4
SwitchAXI-4
StreamSwitch Each AI Engine to NoC can direct traffic between

HNoC and the AXI4-Stream switch


MM AXI4
SwitchAXI-4
StreamSwitchPL
Interface

NoC
i/f

DMA

NSU

NMU

PL
i/f

PL Interface Tile

Aggregation of 32-bit AI Engine stream
into 128-bit NoC stream

HNOC

NSU

128-bits(7.2 GB/s) The number of horizontal and vertical channels inthe device will limit b/w

Programmable Logic

Note: B/W calculation using -1L speedgrade (0.7V), higher bandwidth is available with faster speed grade26© Copyright 2022 AMD

AI Engine Array Interface

NOC Interface Tile

[Public]

Data Movement Architecture: Device Integration

Compute

Mem

K0

K1

Mem

K2

Mem

K3

(AIE Array)

Fabric(Programmable Logic)

I/O

I/O Interface
Logic

GT

Custom On-chip
Memory Hierarchy

PL
Accelerator

Processing
System27© Copyright 2022 AMD

Hard Memory
Controllers

[Public]

AI Engine Delivers High Compute Efficiency
VECTOR PROCESSOREFFICIENCY Adaptable, non-blocking interconnectFlexible data movement architecture
 Avoids interconnect “bottlenecks”

80%

95%

98%

Peak Kernel Theoretical Performance Adaptable memory hierarchy

Local, distributed, shareable = extreme bandwidth No cache misses or data replication
 Extend to PL memory (BRAM, URAM)
 Transfer data while AI Engine Computes
Comm
ComputeComm
ComputeComm
ComputeOverlap Compute and Communication
28© Copyright 2022 AMD

ML C O N V O L U T I O N S

FFT

DPD

Block-based
Matrix Multiplication(32×64) × (64×32)

1024-pt
FFT/iFFT

Volterra-based
forward-path DPD

[Public]

AIE vs AIE-ML
Key differences

29© Copyright 2022 AMD

[Public]

Computed Density/Efficiency Progression

Perf/mm2 & Perf/W+ AI Engine Architecture
improvements & optimization for ML

2-4X
AI/ML
Real-time
5X 5X Signal
AI/ML
Processing

2017/18

AI Engine for
Machine Learning:
AIE-MLReal-time
Signal
Processing*+ AI Engine* Similar performance expected for FFTs.
Some DSP applications might have higher
performances with AIE

2020/21/22

2023/24/25

30© Copyright 2022 AMD

[Public]

Versal Series with AI Engine
Versal® AI Core Series

AIE

AIE-ML

Versal AI Edge Series

AIE-MLAIE

AIE-ML

Versal Premium Series

AIE31© Copyright 2022 AMD

AIE

[Public]Intelligent Engines Optimized for Any AI Application
BEAMFORMING,

CNN, RNN, MLP

RADAR PROCESSING,

 Advanced Signal Processing

HIGH PERF. COMPUTING

Machine Learning 

AIE-MLAIE

AIE Architecture
1X
Compute
Tiles

1X

1X
1X

1X
1X

1X
1X

1X
1X

AIE

OPS / Tile

256

INT4

256

INT8

64

INT16

No native support BFLOAT16

LUTs

AIE-MLAIE-ML Architecture

1024
512
128

256

16

INT32

32*

16

FP32

51*

Compute
2X
Tiles
Mem
Tiles

512KB

2X

2X

2X

2X

512KB

512KB

512KB

512KB

LUTs

*Via software emulation (FP32 not IEEE-754 compliant)UltraRAM
LUTs

 Optimized for ML Inference Applications

KB / Tile

 Optimized for signal processing AND ML
 Flexibility for high performance DSP applications
 Native support for INT8, INT16, INT32, FP32

32

Data
Memory64

16Program
Memory16

 Maximum AI/ML compute with reduced footprint Native support for INT4, INT8, INT16, INT32, bfloat16
 Fine grained sparsity HW optimization
 Enhanced FFT & complex math support

Versal® AIE-ML offers 2X AI Performance per Watt
32
© Copyright 2022 AMD

[Public]Intelligent Engines Optimized for Any AI Application
BEAMFORMING,

CNN, RNN, MLP

RADAR PROCESSING,

 Advanced Signal Processing

HIGH PERF. COMPUTING

Machine Learning 

AIE-MLAIE

AIE Architecture
1X
Compute
Tiles

1X

1X
1X

1X
1X

1X
1X

OPS / Tile

256

INT4

AIE-MLNo native support BFLOAT16

LUTs

1024

256

Reduced data16
INT32
movement

32*

16

51*

UltraRAM

FP32

LUTs

 Flexibility for high performance DSP applications

32

2X AI Data
Perf/W
Memory16

Program
Memory

Mem
Tiles

512KB

2X

2X

2X

2X

512KB

512KB

512KB

512KB

LUTs

 Optimized for ML Inference Applications

KB / Tile

 Optimized for signal processing AND ML

Compute2X
Tiles*Via software emulation (FP32 not IEEE-754 compliant)Reduced AI
PL Footprint Native support for INT8, INT16, INT32, FP32

AIE-ML Architecture

256INT8/16
INT8
2X
OPs/Tile512
128
4X INT4INT16
OPs/Tile
64

1X
1X

AIE

64

 Maximum AI/ML compute with reduced footprint Native support for INT4, INT8, INT16, INT32, bfloat16

16

 Fine grained sparsity HW optimization
 Enhanced FFT & complex math support

Versal® AIE-ML offers 2X AI Performance per Watt
33
© Copyright 2022 AMD

[Public]

AIE
AIE-ML

Multi-Precision Support
Complex Data Types

MACS / CYCLE (PER
CORE)

MACS / CYCLE (PER
CORE)

16

16
8X4
REAL

0

2

4

8

128

8X8
REAL

8

128

32X32 32X32 32X16 16X16SPFP* REAL** REAL REAL

64

32
64

16
32

8
16

8
25

128

256

512

32

Real Data Types

16X8
REAL

32X32
C O MP L E X

32X16
C O MP L E X* 32 SPFP x 32 SPFP emulated using BFLOAT16 (following this paper). Not IEEE-754 compliant. Low accuracy mode**32 Real X 32 Real is emulated

34© Copyright 2022 AMD

16X16
C O MP L E X

1 6 C O MP L E X
X 16 REAL

[Public]

AIE-ML

MemoryAIE-ML

MemoryAIE-ML

MemoryAIE-MLAIE-ML

MemoryAIE-ML Array includes Memory tiles at the

AIE-ML

MemoryCascade Interface

MemoryArray

Memory Interface

bottom of the array
All lines of compute tiles are identical in AIE-ML

Memorytile

However, no direct access to the AIE-ML tiles
 Interface through the AIE-ML memory tiles

AI
EngineAI
EngineAIE
35© Copyright 2022 AMD

MemoryAI
EngineAI
EngineMemoryAI
EngineMemory

AI
EngineAI
EngineMemoryInterfaces with PL and NoC stays the same

AI
EngineMemoryLocal memory is always East
 Accessible neighbor memories are always North, South, and West

Memory

Memoryalways from left to right and top to bottom
Memory

Memorytile

AIE-MLNo snake shape of the cascade interface

Memory

Memorytile

AI
Engine[Public]

Memory Interface

Interfaces

Stream Interface
Cascade Interface

AIE-ML tile has 2 directions for the cascade

input (North and West) and 2 direction for
cascade output (South and East)

256

256

512

32
32

512

256

512

512

AIE-MLengine

256
256

256

256

256
256

256

512

256

Cascade interface width is increased in AIE-ML
AIE-MLto 512-bits in AIE-ML (16 lanes of 32-bit data or 8 lanesof 64-bit data) 384-bits in AIE
Direct AXI4-Stream interface

256

256

256
32
32
32
32
128
128
128
128

384



2 x 32-bit input and 2 x 32-bit output in AIE
 1 x 32-bit input and 1 x 32-bit output in AIE-ML
Stream FIFO only in AIE256

AIE engine

384

256
256
256
256

256

256

256

256

AIE
36
© Copyright 2022 AMD

Stream FIFO



32
32
32
32

[Public]

Core level
Number of multipliers doubled in AIE-ML to 256 (int8 x int8) vs. 128 (int8 xint8) in AIE
Added native support for INT4 and BFLOAT16 in AIE-ML

Removed native support for SPFP32


can be emulated using BFLOAT16 (not IEEE-754 compliant). Emulation following this paperAdded support for 2:4 sparsity in AIE-ML

Removed support for 5G specific extensions in AIE-ML
Removed support for scalar non-linear functions (for example sqrt, Sin, Cos,and InvSqrt) and scalar floating point / integer conversions

37© Copyright 2022 AMD

[Public]

Memory Module
Stream In

Stream Out

38© Copyright 2022 AMD

32

32

32

256
256
256

Locks

AI Engine Interface

AI Engine Interface

AI Engine Interface

Memory Bank

256

256

256

Locks

6-bit unsigned lock state

Memory Bank

Memory Bank

Memory Bank

Memory Bank

locks in AIE

Locks (16)East Memory Port

Arbiters

South Memory Port Semaphore locks in AIE-ML vs. binary

Memory Bank

Locks

Memory Bank

256

Arbiters
Memory Bank

256

West Memory Port

256

MM2S DMA (2)North Memory Port

Improved address generation to support 3D
addressing in AIE-ML
 Supports S2MM finish on TLAST and out-oforder packets
 Compression (MM2S) and decompression(S2MM)

256

S2MM DMA (2)256

 DMA improved in AIE-ML

256

 Memory doubled in AIE-ML to 64KB

Locks

32

AI Engine Interface

[Public]

Memory Interface

AIE-ML Memory Tile

Stream Interface

I/O

PS

AIE-ML Tile

PL

 AIE-ML Memory Tile (AIE-ML architecture only)

AXI4-Stream
Interconnect



512KB memory arranged into 16 banks
 MM2S and S2MM with 6 channels each32-bit stream interfaces128-bit memory interfaces

Support 4D tensor address generation

Supports compression and decompression of the data

AXI4-Stream interconnect with North and Southports (no East and West) 1 or 2 rows at the bottom of the array39© Copyright 2022 AMD

DMA
6xMM2S
6xS2MM

512 KB Memory

Control,
Debug,
Trace

Memory mappedswitchAIE-ML Interface Tile

[Public]

Memory Tile
For Each Bank

9 read interfaces


AXI-MM Read (inc. ctrl-pkts)



MM2S (MEM tile -1)



6x MM2S channels [0-5]MM2S (MEM tile +1)

9 write interfaces


AXI-MM Write (inc. ctrl-pkts)



S2MM (MEM tile -1)



6x S2MM channels [0-5]S2MM (MEM tile +1)

40© Copyright 2022 AMD

[Public]

Memory Tile
For Each Bank

9 read interfaces


AXI-MM Read (inc. ctrl-pkts)



MM2S (MEM tile -1)



6x MM2S channels [0-5]MM2S (MEM tile +1)

9 write interfaces


AXI-MM Write (inc. ctrl-pkts)



S2MM (MEM tile -1)



6x S2MM channels [0-5]S2MM (MEM tile +1)

41© Copyright 2022 AMD

[Public]

AIE / AIE-ML Kernel Performance using AIE APIs
Non-ML kernel performance (cycles) - lower is better
5000
4500
4000
3500

AIE Cycles

3000
2500
2000
1500

1000
500

0
2K FFT
2K FFT(CINT16 x CINT16) (CINT32 x CINT16)

FIR 16t
FIR 32t
Matrix Mult
512 samples
512 samples
32x32x32(CINT16*CINT16) (CINT16*CINT16) (CINT16 x CINT16)

AIE1

Matrix Mult
32x32x32(INT32 x INT32)

AIE2
AIE-ML42© Copyright 2022 AMD

Color conversion Y Color conversion Vertical resizer LUT Vertical resizer
4k pixels
UV
4k pixels
4k pixels(INT16 x INT16)
4k pixels(INT16 x INT16)(INT16 x INT16)(INT16 x INT16)

[Public]

AI Engine vs AI Engine-ML summary
AI Engine

AI Engine-ML

checkerboard

All lines identical

384-bits wide
Horizontal Direction

512-bits wide
Horizontal and Vertical Directions

2*32-bit In and 2*Out 32-bit out

1*32-bit In and 1*Out 32-bit out

512/256 bits

512/256 bits

Int8 * Int8 Multipliers

128

256

Int4 * Int8 Multipliers

128

512

Format Native Support

Int8/16/32, Cint16/32, SPFP32

Int4/8/16, cint16, BFloat

Non-Linear Functions in scaler
processor

Yes

No

Tile Local Memory

32KB

64KB

-

Support for 3D addressing modes
S2MM finish on TLAST and out-of-order packets
Compression/Decompression

BooleanSemaphoreNo

Yes (512KB)

Array structure
Cascade Interface

Tile Stream Interfaces
Memory Load/Store (per cycle)

Tile Local Memory DMA

Local Memory Locks
Memory Tiles

43© Copyright 2022 AMD

[Public]

ML Performance Comparisons
Versal® Devices vs Nvidia Jetson Xavier and TX2
Notes:
1. Jetson benchmark data illustrates
maximum performance results• The maximum throughput
results were obtained
without latency limitationand illustrate the maximum
performance that can beachieved• https://developer.nvidia.com/embedded/jetsonbenchmarks

AI/ML Performance Benchmark Comparisons432
Jetson AGX Xavier

55

29

VGG-19

2,109

704
313

Jetson Xavier NX

1,127

618

YOLOV3-tiny
1,100

ResNet-50

405

29
112
112
2
32

Jetson TX2ResNet-34
Inception-v4
302

VCK190 96 AIEs

23

1,323
1,386

279

VCK190 384
394 AIEs
AIEs
VCK5000

2. Versal AI Core VC1902 device on
VCK190 Eval Board with 96 AIEs
measured with hardware

5,000

16,400

VE2802

VE2302

3,100

VE2102

3. VCK5000 with 384 AIEs
measured with hardware. Batch
size=8, Latency=1.6ms• Nvidia T4: 3,785 img/s
Latency = 2.1ms

1,100
0

1000

2000

3000

4000

44
© Copyright 2022 AMD

5000

Img/s

4. VE2802, VE2302, VE2102
ResNet-50 performance simulated[Public]

Software and Tools

45© Copyright 2022 AMD

[Public]

AI Engine Vitis™ Tool Flow46
© Copyright 2022 AMD

[Public]

Vitis™ Unified Software Platform Guiding Principles
Extensive General andApplication-SpecificLibraries

Single Software
Developer Toolkit
Cloud to Edge

All hardware accelerated

Scalable, Unified AI Flow
For all Xilinx devices

Comprehensive
Technical Content

Vitis

Continually Expanded
and refreshed

Improved Ease of Use
For Software & AI developers

Leverage Ecosystem
Well-defined and documented APIs
Consistent and compatible across releases>> 47© Copyright 2022 AMD

[Public]

Unified Software Platform
Software & AI
Adaptive computing
Edge to Cloud>> 48© Copyright 2022 AMD

[Public]

Vitis Unified Software Platform
For Heterogenous Compute, Edge to Cloud

Domain-specificdevelopment
environments

Vitis™ accelerated
libraries

AI

Vision & Image
Processing

Video Analytics SDK

Blockchain Solution

Model Composer

Math & Linear
Algebra

Graph

HPC

PartnerDevelopment
Environments

Networking

Database…

Data Analytics

Compiler and Linker

Vitis coredevelopment kit

Vitis HLS

RTL Kernel Wizard

AI Engine Compiler

Xilinx runtime library (XRT)

Vitis target
platform

Vitis target platform49© Copyright 2022 AMD

DebuggerAnalyzer[Public]

Development Platforms for ALL Developers
AI & Data
Scientists
Application
SoftwareDevelopers

Embedded / SoC
Developers

HardwareDevelopers

50© Copyright 2022 AMD

[Public]

Vitis™ Platform Philosophy: Platforms and Subsystems
Programmable Logic + AI Engine

Processing System
HW
Platform

Subsystem #1

AIE

PL

SW
Platform

FirmwareSubsystem #2

PL

FirmwareSubsystem #N

PL

FirmwarePS
Application

 Subsystems form the customer’s differentiating logic: AI Engine and PL kernels, operating under the supervision of theprocessing system (PS) Versal platform provides essential infrastructure services (CIPS, programmable NoC, I/Os, OS, drivers…)
 Platform insulates developers from low-level details; lets them focus on application development (SW, PL, or AI Engine)51© Copyright 2022 AMD

[Public]

Vitis™ Development Flow for Versal® ACAP
AI Engine

PL (HLS)

PL (RTL)

Platform

AIE Kernels, Graph

PL Kernels (HLS)RTL Kernels

Vitis HW Platform

AIE Simulation

HLS Cosimulation

RTL Verification

Vitis SW Platform
Linux + rootfs

PS
XRT, Graph API
AIE driver

PS AppPL and AIE Integration (v++ --link)

Vivado HW Build
SIM Build
Timing Closure

Generate Binary (v++ --package)
SSW

Run on Device

ProfileVivado® ML

HW Emulation

Vitis™ Platform

Debug

AIESimQEMU

52© Copyright 2022 AMDSIM

[Public]

AI Engine API(High-Level Intrinsic APIs)

53© Copyright 2022 AMD

[Public]

AI Engine Programming
Application

Kernel Development is Difficult

User codeLong Development Times
 Hard to Maintain Difficult to optimize for performance

AIE
Intrinsics

AIE

Application

Libraries / Frameworks

User codeSpecific Functionality
 Do not address custom kernel development
 Internal library development is still hard and costly

Libraries / Frameworks

AIE
Intrinsics

AIE

54© Copyright 2022 AMD

[Public]

AI Engine API
Application

AI Engine API - Provides an easier to use

programming model through higher-level
abstraction that requires reduced knowledgeof the underlying hardware to achieve
optimized performance

User code
Libraries / Frameworks

AIE
Intrinsics

AIE

Application
User code
Libraries / Frameworks
Libraries /Frameworks
AIE API
AIE
Intrinsics

AIE

55© Copyright 2022 AMD

[Public]

AI Engine API - Portability
Kernel code that is portable across AI Engine derivatives
No need to completely rewrite to take advantage of the latest features

Application w/o AIE API
User code

Application w/ AIE API
User code

Libraries

Libraries / Frameworks
Libraries /Frameworks
AIE APIAIE
ISA/Intrinsics

AIE-ML
ISA/Intrinsics

AIE

AIE-ML…

AIEx
ISA/Intrinsics

AIE
ISA/Intrinsics

AIE-ML
ISA/Intrinsics

AIEx

AIE

AIE-ML56© Copyright 2022 AMD…

AIEx
ISA/Intrinsics

AIEx

[Public]

AI Engine API – Reduced Development time
int16 find_max(const int16 *data, unsigned count){
const v32int16 *pData = (const v32int16 *)data;
v32int16 vec = pData[0];for (unsigned i = 1; i < count / 32; ++i) {vec = max32(vec,0, 0x06040200, 0x0E0C0A08, 0x3210,
pData[i], 32, 0x06040200, 0x0E0C0A08, 0x3210);}
vec = max32(vec, 0, 0x06040200, 0x0, 0x3210, 16, 0x06040200, 0x0, 0x3210);
vec = max32(vec, 0, 0x0200, 0x0, 0x3210, 8, 0x0200, 0x0, 0x3210);
vec = max32(vec, 0, 0x00,
0x0, 0x3210, 4, 0x00,
0x0, 0x3210);
vec = max32(vec, 0, 0x0,
0x0, 0x3210, 2, 0x0,
0x0, 0x10);
vec = max32(vec, 0, 0x0,
0x0, 0x3210, 0, 0x0,
0x0, 0x1);
return ext_elem(vec, 0);}Reduced Development timeEasier to read code
 Easier to maintain code Equivalent performance57© Copyright 2022 AMD

[Public]

AI Engine API vs Intrinsics
When to use AI Engine API vs Intrinsics

AIE API
Reduced Development Cycles

Maintainability
Portability
Performance

✓
✓
✓
✓

Direct Control

58© Copyright 2022 AMD

Intrinsics✓✓

[Public]

Vitis™ Model Composer for AI Engine

59© Copyright 2022 AMD

[Public]

Vitis Model Composer for AI Engine

Vitis™ Development Flow for Versal® ACAPAIE

PL (HLS)

PL (RTL)

Platform

AIE Kernels, Graph

PL Kernels (HLS)RTL Kernels

Vitis HW Platform

AIE Simulation

HLS Cosimulation

RTL Verification

Vitis SW Platform
Linux + rootfs

PS
XRT, Graph API
AIE driver

PS AppPL and AIE Integration (v++ --link)

Vivado HW Build
SIM Build
Timing Closure

Generate Binary (v++ --package)
SSW

Run on Device

ProfileDebug

Vivado® ML

HW Emulation

Vitis™ Platform

AIESimQEMU

60© Copyright 2022 AMDSIM

[Public]

Connectivity Between Domains

HLS Kernel

AI Engine

RTL Kernel(PL)

System Generator functionality
included in Vitis™ Model Composer61© Copyright 2022 AMD

[Public]

Mixed Domain Design: AI Engine + HDL (System Generator)

AI Engine Design
Kernels, Graphs

AI Engine to RTL
Gateway62© Copyright 2022 AMD

RTL Design

RTL to AI Engine
Gateway[Public]

Mixed Domain Design: AI Engine + HLS
HLS Design

AI Engine to HLS
BridgeAI Engine to HLS
BridgeAI Engine Design
Kernels, Graphs
63© Copyright 2022 AMD

[Public]

Model-Based Design – Simulation Support
Heterogeneous Functional Simulation:
AI Engine, HLS, and System Generator blocks all in the same design!

64© Copyright 2022 AMD

[Public]

Model-Based Design – Simulation Support

x86 Simulator
AIE SystemC Simulator

65© Copyright 2022 AMD

[Public]

Hardware Co-Simulation>> 66© Copyright 2022 AMD

[Public]

Code Generation
Graph Code from AI Engine sub-system

HLS C++ code from HLS sub-system

RTL code from System Generator

sub-system

67© Copyright 2022 AMD

[Public]

Model Composer for AI Engine Development
Visualization (connectivity)
graph.h68© Copyright 2022 AMD

[Public]

AI Engine Libraries

69© Copyright 2022 AMD

[Public]

AI Engine Library Overview
Enable short development cycles
Provide pre-packaged kernels & examples
 Quick learning & adoption of AI Engine design Remove kernel level coding for common functions
 Portability across AI Engine architectures (AIE → AIE-ML)
 Allow customers to focus on delivering proprietary algorithms
 Align to Vitis™ acceleration library infrastructure

DSP

Math

Communications(Restricted)

70

Linear Algebra

Vision & Image© Copyright 2022 AMD

Data Movers

[Public]

Vitis™ AI for Versal® ACAP
Machine Learning with AI Engine

71© Copyright 2022 AMD

[Public]

Frameworks for Any Developer
Domain SpecificArchitecture(e.g. AI Inference)
Architecture
Overlay
Data Flow
w/ Xilinx libraries

Kernel Program
Data Flow w/ user
defined libraries

Target Domain Specific Architectures – No HW Design Experience Required72© Copyright 2022 AMD

[Public]

Accelerating AI Inference in the Data Center
User works in Framework of choice1

Deep Learning Frameworks• Develop & train custom network• User provides trained model

Xilinx DNN Compiler implements network

2• Targets AI Inference Domain Specific Architecture• Quantize, merge layers, prune• Compile to AI Engines

Xilinx DNN Compiler
Xilinx AI Inference
Domain Specific Architecture

Scalable across hardware targets

3• Start with Alveo today
Alveo™ U200 / U250

VCK5000

For more information on Vitis AI73© Copyright 2022 AMD

[Public]

Versal® ACAP Design Partition(AI Engine vs DSP and Programmable Logic)

74© Copyright 2022 AMD

[Public]Intelligent Engines in Versal® ACAP → AI and DSP Engines
AI Engine(1) 2D ArrayVLIW and SIMD Architecture
C/C++ Programmable

FPGA Fabric DSP

DSP48E2 Slice

LUT and Memory

Hardened MULT & ADDERS
ACC = ACC + (A  B)

RTL Entry

Increasing
Capability

Increasing
Capability

RTL Entry

DSP Engine

Additional Features
RTL Entry

Increasing
Capability

SPFP32 Native support
75© Copyright 2022 AMD

[Public]

AI Engines and DSP Engines
AI Engine(1) 2D Array

Why AI Engine?

VLIW and SIMD Architecture
C/C++ Programmable Massive compute performance S/W programmable (C/C++) Fast compile – increase productivity

Why DSP Engine?
 Existing RTL/HLS IP usage

DSP Engine

Additional Features
RTL Entry Additional features not available in AI
Engine, e.g. 58-bit Logic Unit
 Pre/Post processing to/from AI Engine

Why both?
 Versal® ACAP accelerates the complete application
 AI Engine efficiency + PL flexibility (inc. DSP)>> 76
© Copyright 2022 AMD

[Public]

Designs with Versal® AI Core Device- DSP & PL v AI Engine
Key System Level Considerations(Design Partition at the system level – not in isolation, i.e. not DSP vs AIE)

ComputeMemoryCompute/Func types• Multiply• Multiply & Accumulate• FFT• FIR etc

Custom Mem Hierarchy• URAMs & BRAMs• LUTRAMs• AIE Data Memory• 32 & 128KB• Maybe external DDR

Data Widths• AIE native 8/16/32 bit• DSP & PL – variable
Vectorizable• Can the dataflow be
vectorize

Type of Mem Access• Random access• Structured v
Unstructured Data
• Data reuse or sharing

Throughput
Latency// Latency

Data
Movement
Movement

Throughput or Latency• AIE can provide both
• Most applications lean
towards one or theother• Decisions onpartitioning likely toaffect thoseparameters and need
to be understood

Data movement key
factor• Data into and out of AI
Engine array• Data around the array(Cascade, AXI etc,)• PL interface for Data
• NoC for trace & debug

77© Copyright 2022 AMD

Data Movement Libraries• Must simplify the
problem for users via
utilities and libraries

Power
AIE brings performance• Perf/watt• Perf/watt/mm2
Managing these key
system levelconsiderations enables
Versal AI Core meet
expectations and deliver
the promised perf/watt[Public]

Representative AI Engine Algorithms
Linear Algebra
Matrix-Matrix Multiplication
Matrix-Vector Multiplication

Convolution
FIR Filters
2-D Filters

Transforms
Fast Fourier Transforms (FFTs/iFFTs)
Discrete Cosine Transforms (DCT)78© Copyright 2022 AMD

[Public]

FIR Filter Micro Benchmarks
Filter 1:• 32-tap real filter• Symmetric• 16-bit data & Coeffs• 1 Gsps input sample rate
Filter 2:• 8-tap real filter• Non-symmetric• 24-bit data & Coeffs• 500 Msps input sample rate

Filter 3:• 32-tap complex filter• 16-bit complex, data & Coeffs• 1 Gsps input sample rate

AIE

DSP58

Processor/Operator Performance

32 GMACs @ 16-bits

0.5 GMACs @ 27x25 bits

Resource Usage
16 GMACs (16 bits real)

0.5 AI Engines

32 DSP58s

AIE

DSP58

Processor/Operator Performance

32 GMACs @ 16-bits
8 GMACs @ 32-bits

0.5 GMACs @ 27x25 bits

Resource Usage
4 GMACs (24 bits real)

0.5 AI Engines

8 DSP58s

AIE

DSP58

Processor/Operator Performance

32 GMACs @ 16-bits (real)
8 GMACs @ 16-bits (complex)

2 DSPs in DSPCPLX slice
0.5 GMACs @ 16x16 bits

Resource Usage
32 GMACs (16 bits complex)

4 AI Engines

64 DSP58s

79© Copyright 2022 AMD

[Public]

Representative Applications

80© Copyright 2022 AMD

[Public]

AI Engine – Market / Application Diversity

Data Center

5G WirelessAerospace & Defence

Wired

ADAS/AD

Test & Measurement

Fintech81© Copyright 2022 AMD

[Public]

Market Requirements and Trends: Wireless 5G
5G Complexity is 100X that of 4G

82© Copyright 2022 AMD

PA, LNA,
Diplexer

Antenna Array

Analogue Radio

Digital Radio
DUC, CFR,
DPD, DDC

ADC / DACTransforms
iFFT/
FFT

SwitchingTransport L2–L7 Modulation IQ
Linear& CTRL& FEC Switch Algebra

Baseband Processing

Beam Forming & MMIO
+ Some Baseband

ETRI RWS-150029,
5G Vision and Enabling Technologies: ETRI Perspective 3GPP RAN Workshop
Phoenix, Dec. 2015
http://www.3gpp.org/ftp/tsg_ran/TSG_RAN/TSGR_70/Docs

Higher Layer Processing

˃ Massive MIMO˃ Multiple antenna, frequency bands
˃ Changing functional partitioning

Packet Processing
and Wired Backhaul

Still Evolving Standard

New Technologies in 5G

[Public]

5G Wireless on Versal® ACAP
Antenna Array

Analogue Radio

ADC / DAC

Digital Radio

Transforms

Beam Forming & MMIO
+ Some Baseband

Switching

Baseband Processing

Higher Layer
Processing

Packet Processing
and Wired Backhaul

5G Wireless Infrastructure (i.e., base-station)
Digital Radiowith ADC/DAC
DPD Update

Compute Maps
to AI Engine

Mapping Example

CPRI

DUC

DPD

ADC/
DAC

AI Engines

Processing
System

I/O
DUC

DPD

Control
Maps to PS

ADC/DAC
DPD Update

Programmable Logic
CPRI

1: DUC: Digital Up Converter
2: DPD: Digital Pre-Distortion
3: Direct RF: ADC/DAC
4: CPRI: Common Public Radio Interface

I/O Maps to PL

83© Copyright 2022 AMD

[Public]

AI Inference on Versal® ACAP
Convolutions

Pooling

Fully Connected Layers

Activations

single depth slice
X

1
4
3
1

0
6
1
2

2
6
1
2

3
8
0
4

y

6
3

8
4

y
yi = xi

yi = xixyi = 0yi = ai xi

Y

ReLU

ReLU/PReLUAI Engines
I/O

Processing
System• Video• Genomics• Storage• Database• Network IPS• Risk modeling(GT, ADC/
DAC)

Programmable Logic

FeatureMap
Data
Volume**Figure credit: https://en.wikipedia.org/wiki/Convolutional_neural_network84© Copyright 2022 AMD

CustomMemoryHierarchy[Public]

AI Inference Mapping on Versal® ACAP

A = Activations
W = Weights

𝐴 × 𝑾𝟎𝟎 + 𝐴01 × 𝑊10 …
𝐴00 𝐴01
𝑊00 𝑊01×= 00
𝐴10 × 𝑾𝟎𝟎 + 𝐴11 × 𝑊10 …
𝐴10 𝐴11
𝑊10 𝑊11
ScalarArm
Dual-Core
Cortex-A72

Adaptable

Intelligent
AI Engines

WeightBufferPlatform
Management
Controller

Fully
Connected
LayersActivation
Buffer(URAM)AI
EngineAI
EngineAI
EngineConvolution
Layers(URAM)

Arm
Dual-Core
Cortex-R5

AI
Engine

Max
Pool

PL

ReLU

X

=

(4x8)

𝐴00 𝑊00 𝐴10

CascadeStream

(4x4)
(8x4)

 Custom memory hierarchy

Programmable Network on Chip

Buffer on-chip vs off-chip; Reduce latency and power

I/O Stream Multi-cast on AI interconnect
External Memory(e.g., DDR)

Weights and Activations



Read once: reduce memory bandwidth AI-optimized vector instructions (128 INT8 mults/cycle)
85© Copyright 2022 AMD

[Public]

Partitioning Example:
Vision Application

86© Copyright 2022 AMD

[Public]

Example Mapping for Vision Application to Versal® AI
Core Device

Analyze

Sense
INT16

FIXED

INT16

INT8

MIPI

Scaling& ISP

HistogramAI

MeasureIntensity Level

Regionof Interest

SensorFusionImage Data

Pixel

Act

INT8

INT8

Floating Point

Mixed

Crop

ObjectClassification(CNN)

Tracking

Act

Feature(LSTM)

Decision

Custom Data-Flow, Custom Precision, Specific Compute87© Copyright 2022 AMD

[Public]

Match Application Components to Heterogenous ACAP
Key is to correctly map into each domain!
INT16

FIXED

INT16

MIPI

Scaling& ISP

HistogramAI

MeasureIntensity Level

Regionof Interest

SensorFusionImage Data

INT8

Heterogenous system

INT8

INT8

Floating Point

Mixed

Crop

ObjectClassification

Tracking

Act

Scalar Engines

Compute engines

FPGA

Arm
Dual-Core
Cortex-R5

DSP

Real-Time
Processor

Adaptable memory

Intelligent Engines

VersalAdaptable
HardwareAI
Engines

Application
Processor

AIE

Adaptable EnginesArm
Dual-Core
Cortex-A72

CPU(LSTM)(CNN)

LUTRAM, BRAM, URAM

Block RAM

Processing System

UltraRAM

Platform
Management
Controller

Accelerator RAM

DSP
Engines

Accelerator RAM

Local AIE Memory

Network On ChipHBM
DDR External MemoryFlexible interconnect

PCIe &
CCIX(w/DMA)88© Copyright 2022 AMD

MIPI

112G

DDR

HBM

58G
28G

Multirate
Ethernet

600G
Cores

LVDS
GPIO

DirectRF[Public]

Match Application Components to Heterogenous ACAP
INT16

FIXED

INT16

INT8

MIPI

Scaling& ISP

HistogramAI

MeasureIntensity Level

Regionof Interest

SensorFusionImage Data

INT8

INT8

Floating Point

Mixed

Crop

ObjectClassification(CNN)

Tracking

Act(LSTM)

Scalar Engines

 Scalar Engines

Adaptable Engines

Intelligent Engines

VersalAdaptable
HardwareAI
EnginesArm
Dual-Core
Cortex-A72

SW programmable sequential processing Complex algorithms and decision making
 Initial configuration and real-time management of
the system Data transfers between DDR and AI Engine array Runtime configuration

Application
Processor

ArmDual-Core
Cortex-R5
Real-Time
Processor

Block RAM

Processing System

UltraRAM

Platform
Management
Controller

DSP
Engines

Accelerator RAM

Network On Chip
PCIe &
CCIX(w/DMA)89© Copyright 2022 AMD

MIPI

112G

DDR

HBM

58G
28G

Multirate
Ethernet

600G
Cores

LVDS
GPIO

DirectRF[Public]

Match Application Components to Heterogenous ACAP
INT16

FIXED

INT16

INT8

MIPI

Scaling& ISP

HistogramAI

MeasureIntensity Level

Regionof Interest

SensorFusionImage Data

INT8

INT8

Floating Point

Mixed

Crop

ObjectClassification(CNN)

Tracking

Act(LSTM)

Scalar Engines Adaptable (PL) & DSP Engines

Flexible parallel compute, custom I/O andmemory Hardware acceleration of compute-intensivefunctions Data processing that exploits heavy data
parallelism
 Deeply pipelined (video codecs),
 Bit manipulations (AES/SHA) Wide datapath Various bitwidth compute

Adaptable Engines

Intelligent Engines

VersalAdaptable
HardwareAI
EnginesArm

HistogramDual-Core

MeasureCortex-A72

Intensity
Level
Application
Processor

ArmDual-Core
Act
Cortex-R5
Real-Time
Processor

Block RAM

Processing System

UltraRAM

Platform
Management
Controller

DSP
Engines

Accelerator RAM

Network On Chip
PCIe &
CCIX(w/DMA)90© Copyright 2022 AMD

MIPI

112G

DDR

HBM

58G
28G

Multirate
Ethernet

600G
Cores

LVDS
GPIO

DirectRF[Public]

Match Application Components to Heterogenous ACAP
INT16

FIXED

INT16

INT8

MIPI

Scaling& ISP

HistogramAI

MeasureIntensity Level

Regionof Interest

SensorFusionImage Data

INT8

INT8

Floating Point

Mixed

Crop

ObjectClassification(CNN)

Tracking

Act(LSTM)

Scalar Engines

 AI Engines

Intelligent EnginesArmAcceleration of vectorized compute intensive
signal processing: Linear Algebra Convolution
 Tensor operations
 Transforms (FFT,DCT)
 8/16/32 bit integers
 Single Precision Floating-Point Real or Complex Operations

Adaptable Engines

HistogramDual-Core

MeasureCortex-A72

Crop

Arm
Dual-Core
Act
Cortex-R5

Versal
Scaling
MIPI Adaptable& ISP
Sensor
Hardware
FusionImage Data

Intensity
Level
Application
Processor

Real-Time
Processor

AI
Engines

Block RAM

Processing System

UltraRAM

Platform
Management
Controller

DSP
Tracking

Engines(LSTM)

Accelerator RAM

Network On Chip
PCIe &
CCIX(w/DMA)91© Copyright 2022 AMD

MIPI

112G

DDR

HBM

58G
28G

Multirate
Ethernet

600G
Cores

LVDS
GPIO

DirectRF[Public]

AI Engine Power & XPE

92© Copyright 2022 AMD

[Public]

AI Engine Power Up to 40X Performance Increase for int8
AIE Power AI Core family -1LL @100C
40

35
30
25
20
15
10
5
0
VC1352 (128 Cores)

VC1502 (248 Cores)

VC1702 (320 Cores)
Dynamic

 Power Per AIE tile 80 to 110mW

Includes memory and interconnectTypical Vector Load is 30% » 70% and Memory R/W rate 10% » 30%

 AI Engines Enable Massive Compute Gains!
93 With only minimal power increase© Copyright 2022 AMD

Static

VC1802 (300 Cores)

VC1902 (400 Cores)

[Public]

Versal ACAP with AI Engine → Power Reduction
 Virtex® UltraScale+™ FPGA 8T 8R 200Mhz -> Versal® ACAP 16T 16R 200Mhz


2x Performance @40% lower power Beamformer Example

2X Performance
200W

80W

94
© Copyright 2022 AMD

50%+ Lower Power[Public]

AIE Power ModelAIE dynamic power is modeled as sum of several components


AI Engine Power (Core and Memory) Interconnect Power PL and programmable NoC Interface Power Top Level Power
Good correlation with silicon when using accurate design parametersInternal correlation based on extracting design info from simulation95© Copyright 2022 AMD

[Public]

AI Engine Power: Engine
AI Engine Power is a function of

Number of cores used in design Kernel type



INT8, INT16, FP

Average “Vector Load”


Vector Load is the portion of time a core spends
performing power-intensive vector operations

This is different from the runtime ratio used by AIE
compiler, which is used for kernel mappingTypical “Vector Load” in internal benchmarks is 30%-70%

For early estimation, the recommendation is to use thedefault of 50%

96© Copyright 2022 AMD

[Public]

AI Engine Power: Local Memory
AI Engine Memory power is a function of

Number of memory banks


Memory banks refers to physical memory banksEach tile has 8 data memory banks

XPE defaults to using all available memory banks

Average “R/W rate”
This is the portion of time when memory bank is accessedwith read or write

100% means every bank is accessed every cycle

If a core reads from a bank every cycle, average R/WRate is 12.5%

Typical Memory R/W Rate is 10%-30%

For early estimation, the recommendation is to use thedefault of 20%

97© Copyright 2022 AMD

[Public]

AI Engine Power: Memory Tiles
For AIE-ML devices
AI Engine Memory power is a function of

Number of memory banks


Memory banks refers to physical memory banksEach memory tile has 16 data memory banks

Average “R/W rate”
This is the portion of time when memory bank is accessedwith read or write

100% means every bank is accessed every cycleFor early estimation, the recommendation is to use thedefault of 20%

98© Copyright 2022 AMD

[Public]

PL and Programmable NoC Interfaces
 Interface power is a function of number ofinterfaces to programmable logic (PL) or
network on chip (NoC)A PL interface is one AXI StreamPower Model assumes 64b

Power error due to 32b or 128b is small

XPE defaults to 1 interface (14 Streams) per 20 Cores

A NoC interface is one NoC AXI interfacePower model does not distinguish between
slave or master interface

Also does not distinguish between memory
mapped or stream interface

Activity on the interface is assumes to be randomdataPower error due to lower or higher activity is
within overall margin99© Copyright 2022 AMD

[Public]

Interconnect PowerInterconnect Power captures power consumesin AXI Stream within the array

Total interconnect power is a function ofInterconnect LoadThis is the percentage of ports used by
the application



Activity on streams is assumes to be randomdata Typical application has Interconnect Load <20%
Value is generated by XPE and cannot be edited

Import flow (from AIE tools data) will use exact values

100© Copyright 2022 AMD

[Public]

Top Level PowerTop level power consists of components that are always on, including

PLL power Shim clock power

Shim does not have coarse-grain clock gating, so regardless of application, it consumes some power

Shim clock power varies with device size (number of columns)

Top-level power in VC1902
0.448 – 0.089 = 0.359W (For a full AIE array <1mW per core)101© Copyright 2022 AMD

[Public]

AI Engine Power Analysis in XPE
To enable the AI Engine Tab, select a
device which includes an AI Engine array(AIE or AIE-ML)

XPE includes tooltip to help users enter
their design’s information

102© Copyright 2022 AMD

[Public]

3 Different FlowsManual Flow

Users enter manually all the information in the AI Engine tabComplier Based Flow

Users import the .xpe file generated by aiecompiler in <build directory>/Work/reports

Simulation Based Flow

Users need to run simulation with VCD file output enabled VCD file can be converted to .xpe file using vcdanalyze command

vcdanalyze --vcd <vcdfile> --xpe.xpe file can then be imported into XPE

103© Copyright 2022 AMD

[Public]

Competitive104© Copyright 2022 AMD

[Public]

Versal® AI Core Series Peak Compute Capacity (int8)
VC1902

VC2602* VC2802*

250

200

*VC2602 and VC2802 have an AIEML array. Other devices of the series
have an AIE arrayInt8 Tops

150

100

Notes:
1. AI Engine @ 1.3 GHz50

128 int8 MACs per AIE
256 int8 MACs per AIE-ML

2.

DSP Engine @ 1.15 GHz
3 int8 MACs per DSP

0

https://www.xilinx.com/support/documentation/selection-guides/versal-ai-core-product-selection-guide.pdf
105© Copyright 2022 AMD

[Public]

Versal® Premium Series vs. Agilex Compute Comparison▸ Peak compute advantage (DSP58 + AI Engine)▸ INT8:
2.8X advantage▸ CINT16: 2X advantage▸ FP32:
1.7X advantage

RF Signal Processing(CINT16)

Radar Data Cube Processing(FP32)

3.3X

2X

4.7X

1.7X

VU13P

AGI040

VU13P

AGI040AI/ML Inference(INT8)

5.8X

2.8X

VU13P

AGI040▸ Agilex increases DSP 50% per LE over Stratix 10▸ No direct response to AI Engine▸ Addressing compute with more of same/similar▸ 9x9 multipliers return to action (not in Stratix 10)▸ Clearly optimizing for DC / ML with new:
▸ Int8 – ML inference optimization▸ FP16 / BFloat16 – ML training and inference EoU▸ A&D & Wireless markets also targeted:▸ Retaining features like Complex MAC▸ Fmax reductions vs. Stratix 10 are evident

VP2802

VP2802

VP2802

Remember: TOPs are only a first order metric for comparing
relative capability. Application-level performance is what reallymatters and future benchmarks will be provided in these areas.

1: AMD-Xilinx results based on calculated compute capacity with DSP slices in Virtex UltraScale+ VU13P FPGA
2: Intel results based on calculated compute capacity with DSP blocks in Intel Agilex AGI040 FPGA and frequency using the fastest speed grade from Intel Agilex Device Data Sheet (DS-1060)
3: AMD-Xilinx results based on calculated compute capacity combined AI Engines and DSP Engines in the Versal Premium VP2802 ACAP>> 106© Copyright 2022 AMD

[Public]

Versal® AI Edge Series: Up to 4X Performance/Watt vs.
GPUs
CPU Accelerator(10-15 Watts)(7-10 Watts)(26-45 Watts)

1.3X

3.3X

4.2X

Performance/Watt

Performance/Watt

Performance/Watt

JetsonXavierJetsonNX1 NX
XavierVersal AI Edge
VersalEdge
VE2102
Device(VE2102)

batch=8
ResNet50 224x224

107

Images/sec/wattImages/sec/wattImages/sec/watt

Intelligent Edge Sensor

Autonomous System
or Edge Aggregation

JetsonAGX
JetsonXavierAGX
Xavier…
(15W Mode)2

Versal AI Edge
VE2302 Device

ResNet50 224x224, batch=1

1: Jetson NX Xavier: https://developer.nvidia.com/embedded/jetson-benchmarks, batch size and number not published
2: Jetson AGX Xavier: https://developer.nvidia.com/embedded/jetson-agx-xavier-dl-inference-benchmarks,
3: Jetson AGX Xavier MAX N-Mode and VE2802 represent the highest performing device configuration in their respective portfolios
Jetson AGX Xavier device power estimated by subtracting published memory & IO power from total module power© Copyright 2022 AMD

Jetson AGX
AGXXavierXavier…

Versal AI Edge
VE2802 Device(MAX N-Mode)2,3
ResNet50 224x224, batch=1

[Public]

Summary108© Copyright 2022 AMD

[Public]

AI Engine Architecture Summary
ComputeEfficiency

8x increase in compute density
At ~40% lower power

HeterogeneousArchitectureHigh-throughput, low-latency
PL flexibility – accelerate the complete application
Custom memory hierarchy

Multiple
Applications

Data center: AI inference
Comms: wireless 5G – radio (DFE), baseband, Wired – cable access
ADAS/AD: embedded vision

Software
Programmable

Frameworks; C/C++
SW compile, execute, debug
Optimized S/W libraries>> 109© Copyright 2022 AMD

[Public]Thank You© Copyright 2022 AMD

[Public]

Disclaimer and Attribution
The information contained herein is for informational purposes only and is subject to change without notice. While every precaution has been taken in the
preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise
correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of thisdocument, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect
to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual
property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement
between the parties or in AMD's Standard Terms and Conditions of Sale. GD-18© Copyright 2021 Advanced Micro Devices, Inc. All rights reserved. Xilinx, the Xilinx logo, AMD, the AMD Arrow logo, Alveo, Artix, Kintex, Kria, Spartan, Versal,
Vitis, Virtex, Vivado, Zynq, and other designated brands included herein are trademarks of Advanced Micro Devices, Inc. Other product names used in thispublication are for identification purposes only and may be trademarks of their respective companies.

© Copyright 2022 AMD