REDUCING DATACENTER POWER CONSUMPTION BY ELIMINATING MEMORY BOTTLENECKS

WHITEPAPER

The Era Of General
Acceleration Has Begun

The Era Of General Acceleration Has Begun

Executive Summary

The era of general purpose computing began in April 1964 withthe launch of the aptly named System/360 mainframe by IBM.
Computing was simpler then: Batch processing of integer data and,eventually, online transaction processing.

But the idea was that one family of systems could support
a whole stack of systems software, including operating
systems, databases, transaction monitors, programming
tools, and such.
Thanks to the System/360’s enormous success, soon

In this new era, computeengines are going toproliferate

thereafter dozens of different minicomputer makers stormed
the datacenter with their own unique processors andsoftware stacks tuned for them, aimed at specific workloads

In this new era, compute engines are going to proliferate,– some of them back office work like the System/360

offering varying types and capacities of compute andmainframes did, some of them focused on new eras such as

memory to match a set of workloads that is diversifying

simulation and modeling.

faster than the SKU stacks of the major CPU vendors. It has
been decades since just being able to manipulate integers

The RISC/Unix revolution vanquished most of thesewas sufficient in a compute architecture, which saw the riseproprietary systems, and the advent of the Pentium Pro andof outboard math coprocessors and then integrated vector

Xeon server processors from Intel coincided with the rise ofengines and now, with the advent of artificial intelligence,the commercial Internet to foster the first nearly universal

the rise of matrix math units to accelerate AI inferencecomputing substrate that was affordable and that could run

and in some cases AI training, too. But all kinds of otherhyperscale workloads that the world had been imagining forfunctions that might otherwise be done by CPU cores are

years.

being done by accelerators, often integrated on cores,
sometimes integrated in the CPU package and other times in

The System/360 in the 1960s and the 1970s, the Unix

outboard accelerators such as GPUs, FPGAs, or DPUs.

systems of the 1980s and the 1990s, and the X86 platformof the 1990s and 2000s represent three different epochs

And importantly, for each workload, the right kind andin the era of general purpose computing. But today,

capacities of compute have to be brought to bear in asomething different is happening. Driven by the need tosystem that meets the physical and economic requirements

highly tune workloads to systems, and to meet very tightof where that workload is running. The era of general

power, thermal, and monetary budgets and to deliver the

purpose computing is dead, and so is the era of the 1U andmost optimal performance and value for the dollar, we are

2U pizza box server. Both the compute engines and theentering what we might call the era of general acceleration.

systems that wrap around them have to be co-designed andoptimized along with the workload software stack because
the end of Moore’s Law demands this.

WHITEPAPER

2

The Era Of General Acceleration Has Begun

Acceleration Is Core To The Future Of
Compute

It is a difficult and delicate
dance, figuring out where

The ability to shrink transistors over the past five decades
made modern computing possible. But system architecture

to compute what in a

is not as simple as putting an entire motherboard on a single
piece of silicon that fits into a single server socket. The fact

market that has seen athat those transistors are getting more expensive as they

Cambrian explosion in

keep shrinking now means that some tough architectural

compute engines.

choices need to be made, and every possible function whichmight be needed cannot be reasonably crammed into thatcore anymore.
Over the past decade, Moore’s Law roughly equated to

The reason is simple: efficiency. Sometimes, when a function

transistors getting cheaper by a factor of 2X every 18 to 24

becomes ubiquitous enough, it makes sense to implementmonths meaning manufacturers could afford to put more

it in an ASIC to drive performance and price/performanceof them on a device. That has begun to slow however whilerather than implement it in software, where its performance

Dennard scaling – as manufacturing process shrinks, youwill be bound by the number cores you can allocate to it andcan boost the clock speed to add more performance at thethe clock speed of those cores. The latter approach uses up

same thermals – has long since stopped. All of which means

valuable CPU cores and burns a lot more electricity per unit ofprocessor and system architects have been struggling withwork.

where to put what kind of compute.
All things being equal, it would have been easier on everyone
Because variety is the spice of life, there is a progressive

if everything could have been implemented in high level

offloading of different compute functions from the coresoftware and run on a more general purpose X86 processor.

to the die or socket to the external data processing

This certainly happened during the Dot Com Boom, thanks

unit (DPU), graphics processing unit (GPU) and otherin large part to shrinking transistors and scaling clock

accelerators outside of the socket. But not every CPU

frequencies on CPUs.

maker draws the lines in the same way while all CPU makers
let their customers, who have very different applications

But because Dennard scaling is dead and Moore’s Law isand therefore very different system needs, use different

limping along, it means we have to rely on an expandingarchitectures to do this progressive offloading. The end

hierarchy of compute to wring efficiencies out of systems.

result is that systems tend to be tailored as much as is

And therefore, our old habits of counting cores and clock

possible in a semiconductor industry that requires a certain

speeds and measuring the relative increase in the integerlevel of volume economics for anything to be built in the first

performance of the cores on a processor, and then picking aplace.

pizza box server after the fact is no longer a good way to builda system.

It is a difficult and delicate dance, figuring out where
to compute what in a market that has seen a CambrianNow, we have to start with the workload, pick the appropriate

explosion in compute engines - there’s an increasing

form factor for the system, including its power, thermal, anddiversity of compute engine makers but an increasing

physical constraints (the machine may not be in an idealized

diversity of instructions within a socket too.

datacenter environment, but somewhere out on the edge),
and then optimize the compute, networking, and storage

WHITEPAPER

3

The Era Of General Acceleration Has Begun

within that system along performance, price/performance,but on the CPUs themselves. Intel is first out of the gate toor performance/watt vectors. The process is a co-design

accelerate inference (and also small-scale AI training) with theof hardware and software within a physical and budgetary

Advanced Matrix Extension (AMX) matrix math units in theenvironment. And you can’t optimize for all of these

Sapphire Rapids CPUs.

constraints at the same time. You have to pick, and it always
involves making tradeoffs.

For application serving including AI inference, the Supermicro
X13 Hyper SuperServer (SYS-121H-TNR) is a good platform

Let’s walk through three different workloads running on

on which to build.

various Supermicro systems that employ Intel’s latest“Sapphire Rapids” 4th Generation Xeon SP processors to see

SUPERMICRO X13 HYPER

how this works. Then, we will dive into the feeds and speedsof all of the Intel Acceleration Engines in the Sapphire Rapids
CPUs that can be brought to bear to boost the performanceof all kinds of workloads.

AI Inference

Artificial intelligence istaking the datacenter bystorm, and every company
is looking at the best way –and the cheapest way – toweave inference into theirworkloads.

The Supermicro X13 Hyper SuperServer system (SYS-121H-TNR)
This machine comes in a 2U, rack-mounted form factor thathas been the standard for application serving in the enterprise
datacenter for two decades now, and it comes with a pair of
Sapphire Rapids processors, 32 DDR5 main memory slots,
room for a pair of internal NVM-Express M.2 flash or SATA3
disk drives for the operating system, slots for eight 2.5-inch
NVM-Express, SAS, or SATA storage devices, and eightinternal fans to keep it all cool. The X13 Hyper SuperServer
also comes with two Advanced I/O Module (AIOM)
networking ports with a variety of Ethernet speeds, a pair of
1,200 watt Titanium-level energy efficient power supplies, andArtificial intelligence is taking the datacenter by storm, anda dedicated baseboard management controller with its own

every company is looking at the best way – and the cheapest

network interface; it also has a bezel-free configuration if you

way – to weave inference into their workloads. They can add

do not want to pay for unnecessary plastic.

custom ASIC or GPU accelerators from various sources totheir application servers, or buy banks of them and offload

The X13 Hyper SuperServer is able to handle Sapphire Rapids

inference from their application servers entirely. But forCPUs rated at up to 350 watts, which means you can put in

latency, security, and cost reasons, these are no longer

low, middle, or top bin parts depending on the applicationnecessarily the best choices.

needs. The performance of the AMX units in the processors
will scale more or less linearly with the number of sockets and

As CPUs get matrix math units that can accelerate AI

cores because there is one AMX unit per core. Core for coreinference, it makes sense to consider keeping inference notagainst the other current X86 server chip in the market, if

only within the same servers that are running the applications,

you do image recognition using the ResNet50 model, natural

WHITEPAPER

4

The Era Of General Acceleration Has Begun

language processing using the BERT Large model, or theprimitives and CRC calculations. It also works with another

DLRM PyTorch recommendation model, you will get a factor

set of algorithms called Quick Assist Technology, or QAT,of 3X improvement in performance on AI inference work overto decrease the memory capacity and memory bandwidththat alternative. And the cost is effectively zero comparedrequirements for analytics and database workloads.

to using outboard accelerators because the AMX units areincluded in the Sapphire Rapids cores essentially for free.

The jump in performance per socket for the Sapphire Rapids
chips compared to the prior generation “Ice Lake” 3rd Gen
Xeon SPs is significant. With the Clickhouse DB database,

Database Acceleration

performance is 59 percent higher on a 60-core Xeon SP8490H than on a 40-core Ice Lake Xeon SP-8380 machine,

Now, let’s build a fast database server. Here, we will start

and with the RocksDB database, performance is up to 3X

with the Supermicro X13 Multi-Processor server, (SYS-

higher compared to the Ice Lake equivalent. By using the

241H-TNRTTP), which scales up to four-way configurations

accelerators, that means a database server like the Supermicro

and is designed for heavy database workloads where lots of

X13 Multi-Processor system could do that much more work,

compute, storage, and memory are brought to bear to support

or companies could cut back on the cores they deploy toa single instance of a database management system.

get the same performance and possibly radically lower their
database licensing costs.

SUPERMICRO X13 MULTI-PROCESSOR
In this case, customers could optimize for both performanceand price, which is a rare thing indeed in the datacenter.

5G Edge Workloads
As our smartphones become the most important personal
computing device in our lives and as AI and other workloads
move out closer and closer to them to provide the lowest
possible latency for those workloads, systems have to be
The Supermicro X13 Multi-Processor system (SYS-241H-TNRTTP)

The X13 Multi-Processor system comes in a 2U rack-mounted
form factor that can have up to four of the Sapphire Rapids
CPUs. It has 64 DDR5 memory slots, with a maximum

deployed in somewhat hostile environments out at the edge.
For such workloads, the Supermicro X13 SuperEdge system(SYS-211SE-31D) is the platform on which to start building.
SUPERMICRO X13 SUPEREDGE

capacity of 16TB using 256GB DIMMs. The system has apair of M.2 NVM-Express or SATA3 devices for the operating
system and eight 2.5-inch bays that can hold flash or disk
drives, and has a pair of AIOM network interfaces and a pairof 2,700 watt Titanium-level power supplies to juice it all up.
With the Sapphire Rapids CPUs, Intel is announcing a new
set of circuits called the In-Memory Analytics Accelerator,or IAA for short, that boosts the performance of analytics
The Supermicro X13 SuperEdge (SYS-211SE-31D)

WHITEPAPER

5

The Era Of General Acceleration Has Begun

The X13 SuperEdge server is a half-depth, 2U rack-mounted
server designed explicitly for telcos and other service

The core is no longer the universal unit of performance in theproviders. It has three hot pluggable, single-socket server

datacenter that it has been over the past two decades. Morenodes that can slide into its enclosure to provide compute andand more functions that were previously outside of the socket

storage for telecom DRAN, CRAN, Flex-RAN, and Open vRAN

have been added to the core – things like PCI controllers,and all kinds of edge applications. The server nodes support

memory controllers, and floating point vector math units toa single Sapphire Rapids CPU rated at between 85 watts andaccelerate HPC. And now, we see CPUs getting matrix math

300 watts, up to 2 TB of DDR5 memory across eight DIMMs,

units to accelerate AI inference and sometimes AI retraining

a pair NVM-Express M.2 flash drives for the operating system

workloads too.

and local data, and three PCI-Express slots for network andother peripheral expansion. The chassis has a pair of 2,000

These present fixed ratios of functions relative to integer

watt power supplies.

performance in the cores. But for other kinds of acceleration– data encryption, data compression, network and storage

With the “Golden Cove” cores used in Sapphire Rapids CPUs,

acceleration, and so forth– putting all of this onto each core

the AVX-512 unit has been tuned to accelerate the vRAN

would be overkill and would actually limit the number of cores

software stack that does the conversion of analog radiothat could be put on a chip to do real work.

signals to digital forms. That means they can be transmitted
over the backhaul in the fiber optic networks that link base

Having integrated accelerator engines does not necessarily

stations to datacenters and their service, for example. This ismean customers won’t implement certain functions outsidedone with Intel’s FlexRAN and Data Plane Development Kitof the CPU package or in software. The necessity of precisely(DPDK) software, which runs certain functions on the AVX-

tuning hardware to the needs of a specific application is going

512 unit if it sees that it is a Sapphire Rapids CPU. With thisto drive complexity across the entire server base, and there

Open vRAN acceleration, this part of the software stack used

is no way out of it as long as the market demands flexibility.

in 5G mobile networks can be boosted by up to 2X. This canWhilst this seems unavoidable, with a slight shift in thinking,potentially mean putting fewer systems out at the edge for 5G

the complexity that enables that flexibility is also desirable.

networks.

So, expect certain functions to be embedded in the CPU
cores, on the CPU packages, in the IPUs and SmartNICs, andin other peripheral cards attached over the PCI-Express bus,

Drilling Down Into The Accelerator
Enginesand in many cases using memory coherence over the CXL
protocol.
When you look at how the applications are actually running,

The core is no longer
the universal unit of
performance in the
datacenter that it has

some functions should be outside of the core and shared bya mesh of cores, but still be placed on the die to ensure low
latency and high performance. It really comes down to how
important a function that was implemented in higher levelsoftware becomes to datacenter hosting operations.

been over the past two

If the function implemented in software consumes a lot of

decades.

CPU cycles, it will be used when necessary but sparingly. This
was the case for data encryption, for more than a decade.
Basically until enough transistors were in the processor

WHITEPAPER

6

The Era Of General Acceleration Has Begun

And so, encryption acceleration came to server CPUs andSuddenly it became
technically andeconomically feasible tostart encrypting all data
at rest and in motion,
without a performance
penalty.

suddenly it became technically and economically feasible
to start encrypting all data at rest and in motion, without aperformance penalty. Putting data encryption acceleratorsin the server CPU package transformed it from a genericsoftware cost that burned a lot of cores or that required
an external accelerator hanging off the PCI-Express bus tobeing included in the server CPU architecture. And witha minor incremental cost or none at all compared to prior
CPU generations and freed up the cores that were runningencryption algorithms.
It is hard to beat inexpensive acceleration embedded on
the CPU chiplet in terms of latency, and you can’t beat‘included for free’ in terms of price/performance. And there

budget and there was enough customer demand – particularly

is a multiplicative effect as software functions are takenfrom the hyperscalers and cloud builders which were mostoff the cores, freeing them to do work, and functions aredefinitely not encrypting any data internally within theirimplemented in a relatively small number of transistors thatdatacenters, but at the same time absolutely knew they

perform those functions much faster.

needed to do so because they were building shared computeutilities.

There are more than a dozen Accelerator Engines in themost recent 4th Gen Intel® Xeon® Scalable processors, andthe number has been gradually increased in the past four
processor generations:SUPERMICRO WP ACCELERATOR ENGINE GENERATION TABLE

WHITEPAPER

7

The Era Of General Acceleration Has Begun

the accelerators and the cores is no worse than between the
cores on the die.
These Accelerator Engines are part of the same coherent,

The six most important of the Accelerator Engines, according

shared memory space as the “Golden Cove” cores on the
Sapphire Rapids package, which means the latency between
SUPERMICRO ACCELERATOR ENGINE SIX KEY

to Intel, that are on the Sapphire Rapids processors, are the

For AMX, Intel is working with the upstream AI frameworks,

following:

toolkits, and libraries – starting with TensorFlow and PyTorch– as well as adding support for AMX in its own oneDNN

The AVX-512 vector and AMX matrix math units are on eachneural network. In time, we believe that AMX could be usedcore in the Xeon architecture.

to support certain kinds of low-precision HPC calculations, we
also believe that certain portions of HPC solvers will be basedon AI models rather than brute force calculations to further

SUPERMICRO ACCELERATOR ENGINE SIX KEY

WHITEPAPER

8

The Era Of General Acceleration Has Begun

accelerate HPC performance. These are works in progress at

The same is true of the other Accelerator Engines in thethe moment.

Sapphire Rapids processors. The idea is to make this asseamless and invisible as possible.

Quick Assist Technology, or QAT, is new to the Xeon SP
processor complex with Sapphire Rapids. What’s interesting

The other three major new accelerators going clockwise –is that Intel has been moving hardware based QATzip data

Dynamic Load Balancer, In-Memory Analytics Accelerator, andcompression and SSL encryption algorithms upstream towards

Data Streaming Accelerator – are brand new with Sapphire

the CPU package over the last fifteen years. QAT started out

Rapids CPUs. Let’s walk through these.

on Ethernet network interface cards way back in 2007, andthen moved onto certain server chipsets, then into chipsets•

Dynamic Load Balancer (DLB): DLB is addressedco-packaged with Xeon D chips, and now is on the chiplets

through the Intel Data Mover Library and it is used(but not the Golden Cove cores) in the Sapphire Rapids

to dynamically redistribute the flow of data across apackage.

core complex and its network interfaces in place ofan

QAT is useful for speeding up and securing distributed

ingress gateway that runs in software. DLB canstorage systems, file systems, databases (with support for theunderpin IPSec security gateways, VPP routers,

open source RocksDB being highlighted), Hadoop and Sparkvirtual switching, streaming data processing, and thedata analytics, NGINX Web application serving, and such.

handling of elephant flows in network function

Customers who create their own middleware and databasevirtualization (NFV) workloads. DLB reduced thesoftware can access QAT acceleration using instructions,latency of load balancing across a microservices

just like any other feature of the Xeon SP CPU. But Intel

workload by 96 percent at the same throughput

has worked with third party software providers and open

compared to using the Istio ingress gateway.
In-Memory Analytics Accelerator (IAA): IAA 		

source projects to make sure QAT is supported wherever it is•appropriate so customers don’t have to think of it.

accelerates analytics primitives and CRC calculations

SUPERMICRO ACCELERATOR ENGINE FOUR NEW

WHITEPAPER

9

The Era Of General Acceleration Has Begun

and works with QAT to decrease the memory		capacity and memory bandwidth

streaming data movement and data transformation
operations that are common in storage, networking,requirements for 		

analytics and database workloads

and analytics workloads. It is accessed through the– particularly 		

in-memory databases, but also

Intel Data Mover Library. On a top-end 60-core

relational databases 		

and datastores – andSapphire Rapids part, DSA was able to push 1.7X

helps to boost the throughput 		

of these systemhigher I/O operations per second on a benchmark

programs. IAA is enabled through 		the combination

juggling large packet sequential reads compared toof Intel’s Query Processing Library 		and its Datarunning Intel’s own Intelligent Storage Acceleration

Mover Library, and is certified to 			accelerate RocksDB, Redis, Cassandra, MySQL,

Library (ISA-L) software on the Sapphire Rapids
cores 		themselves.

and MongoDB as well as several commercial 		•

databases. On RocksDB benchmarks, turning on IAAIntel has not been precise about how much area theseboosted performance by 2.1X compared to not

accelerators take up. But collectively and not including thehaving it on and using Zstd software, which was

AMX and AVX-512 units on the Golden Cove cores, thecreated by Facebook and which is the reference

Accelerator Engine die area looks to be a little bit bigger thanstandard in C, to do compression.

a PCI-Express controller, which is not very big at all.

Data Streaming Accelerator (DSA): DSA optimizes

SUPERMICRO ACCELERATOR ENGINE WORKLOAD

Having the Accelerator
Engines turned off when
they are not needed is as
important as having themturned on when they are.

As you can see from the IAA case above, it is rarely the casethat these Accelerator Engines work alone. They often work incollaboration with each other, and this is how Intel organizes
them as grouped functions for accelerating specific kinds ofworkloads:
Like incandescent light bulbs in days gone by, having the
Accelerator Engines turned off when they are not needed is as
important as having them turned on when they are. Not every

WHITEPAPER

10The Era Of General Acceleration Has Begun

SUPERMICRO INTEL ACCELERATOR ENGINE SERVER TABLE

workload will require acceleration, or all kinds of acceleration.

Sapphire Rapids family of CPUs for example, which are

In some cases, there should be – and will be – variants of

the ones with HBM2e memory support included, the fourCPUs that don’t have some or all accelerated functions

Accelerator Engine tiles are actually replaced with HBM

included in the architecture, or do not have them activated if

memory controllers, so only the AMX, AVX-512, and DSA

they are included in the architecture.

accelerators that are located in the cores are included.

For instance, we fully expect for some CPUs to not have

And in some cases, the Intel Acceleration Engines arevector math units or matrix math units in their cores foravailable through an On Demand activation mechanism,

customers whose workloads do not either require thesewhich means customers only turn them on and pay for them

functions, or they want to offload them to beefier accelerators

when they actually use them.

outside of the CPU. With the Max Series CPUs in theSUPERMICRO ACCELERATOR ENGINE WORKLOAD

WHITEPAPER

Where The Accelerator Rubber Hits
The Server Roads

11

The Era Of General Acceleration Has Begun

data types. It is ideal for workloads like natural-language
Using the Sapphire Rapids in-socket accelerators improves

processing, image recognition and recommendation systems.

performance and performance per watt, and often by factorsof 10X or higher, compared to the prior “Ice Lake” Xeon SPs.

Additionally, AI is an emerging use case on VMware vSAN,

Exactly how much depends on the workload:

and the speed with which AI and deep learning (DL) are
evolving means they will soon be built into nearly every

In the chart above, the baseline is a Sapphire Rapids machineenterprise application and analytics tool. Supermicrowith no acceleration turned on. The performance per wattand Intel have partnered to bring to market an enterprise

improvements are shown for those with the appropriate

application solution for VMware vSAN that is optimized

Acceleration Engines turned on. The wattage of theby latest generation technologies. X13 SYS-221BT-DNTR

accelerators, with the exception of the AVX-512 and AMX

systems with 4th Gen Intel® Xeon® Scalable processors

units, is nominal, which means the performance increase isfeature AMX which gives AI-enabled applications the ability

quite significant in many cases.

to deliver flexible and efficient performance.

Sponsored by Supermicro.

There is an insatiableappetite to use AI to
execute faster andFor more details, visit
www.supermicro.com/en/featured/accelerate-everything

bring new insights andinnovations to industries
to completely disrupt
traditional businessmodels, products andservices.

There is an insatiable appetite to use AI to execute fasterand bring new insights and innovations to industries tocompletely disrupt traditional business models, products and

ABOUT SUPERMICRO

services. Intel has chosen to answer customer demand forSupermicro is a global technology leader committed tomore compute and greater efficiency by offloading specificdelivering first-to-market innovation for Enterprise, Cloud,workload tasks to power efficient accelerator engines. Doing

AI, Metaverse, and 5G Telco/Edge IT Infrastructure. We

this frees up the CPU to do general compute operations

are a Rack-Scale Total IT Solutions provider that designs

which in turn increases overall performance. Intel Advanced

and builds an environmentally friendly and energy-saving

Matrix Extensions (AMX) is a new built-in accelerator

portfolio of servers, storage systems, switches, software,

designed to improve the performance of deep-learning

along with global support services.

training and inference on the CPU for INT8 and BF16

WHITEPAPER

12