Hardware for Deep Learning. Part 4: ASIC
By: Grigory Sapunov
Tags: Hardware, Artificial Intelligence, Tpu, Deep Learning, Asic

References:

AWS

Amazon has its own solutions for both training and inference.

AWS Inferentia

AWS Inferentia was announced in November 2018. It was designed by Annapurna Labs, a subsidiary of Amazon.

Each AWS Inferentia chip contains four NeuronCores. Each NeuronCore implements a high-performance systolic array matrix multiply engine (as Google TPU). NeuronCores are also equipped with a large on-chip cache (but the exact numbers are unknown). [source]

AWS Inferentia supports FP16, BF16, and INT8 data types. Furthermore, Inferentia can take a 32-bit trained model and automatically run it at the speed of a 16-bit model using BF16.

Each chip can deliver 64 TFLOPS on FP16 and BF16, and 128 TOPS on INT8 data. (source)

You can have up to 16 Inferentia chips per EC2 Inf1 instance. Inferentia is optimized for maximizing throughput for small batch sizes, which is beneficial for applications that have strict latency requirements.

The AWS Neuron SDK consists of a compiler, run-time, and profiling tools. It enables complex neural net models, created and trained in popular frameworks such as TensorFlow, PyTorch, and MXNet, to be executed using Inf1 instances. AWS Neuron also supports the ability to split large models for execution across multiple Inferentia chips using a high-speed physical chip-to-chip interconnect.

The technical details on Inferentia are very scarce.

References:

AWS Trainium

December 1st, 2020 Amazon announced its AWS Trainium chip.

AWS Trainium is the second custom machine learning chip designed by AWS and it’s targeted at training models in the cloud.

AWS Trainium shares the same AWS Neuron SDK as AWS Inferentia, so it’s integrated with TensorFlow, PyTorch, and MXNet.

AWS Trainium will be available in 2021.

For now, almost no technical details are available.

References:

Huawei Ascend

Huawei has its own solutions for both training and inference as well. The lineage of AI products is pretty vast, but we’ll focus on accelerator cards mostly.

Atlas 300I Inference Card

The card is built with the Ascend 310 AI processor. The processor delivers 22 TOPS INT8 and 11 TFLOPS FP16 with 8W of power consumption.

The card is a PCIe x16 Gen3.0 device with a maximum consumption of 67 W. It supports one 100 Gbit/s RoCE high-speed port. The card contains 32 GB LPDDR4X with a bandwidth of 204.8 GB/s. A single card provides up to 88 TOPS INT8 computing power.

The pricing is not public now, you have to request the information.

There are also Atlas 800 Inference Servers with up to 7 or 8 Atlas 300I inference cards for AI inference in data centers.

And there are Atlas 200 AI Accelerator Module with the Ascend 310 AI processor to implement image recognition and image classification on the device side, and Atlas 200 DK AI Developer Kit with the Ascend 310 AI processor to facilitate quick development and verification.

Atlas 200 Development Kit

References:

Atlas 300T Training Card

The training card is based on the Ascend 910 AI processor with 32 built-in Da Vinci AI Cores and 16 TaiShan Cores. The processor delivers 320 TFLOPS FP16 and 640 TOPS INT8 of computing performance with 310W of max power consumption. It’s pretty close to NVIDIA’s A100 BF16 peak performance of 312 TFLOPS. Other sources say it delivers up to 256 TFLOPS FP16/512 TOPS INT8.

Ascend 910 is a high-integration SoC processor. In addition to the Da Vinci AI cores, it integrates CPUs, DVPP, and Task Scheduler.

A single card provides 220 TFLOPS FP16 or 280 TFLOPS FP16 computing power in Pro version (?). These numbers are somehow lower than the corresponding numbers of the Ascend 910 processor. It has 32 GB HBM or 16GB DDR4 2933. This card is a PCIe x16 Gen4.0 device with up to 300W power consumption.

HCCS is Huawei’s in-house high-speed interface interconnecting Ascend 910s. On-chip RoCE interconnects nodes directly. PCIe 4.0 doubles the throughput of the previous generation.

The pricing is not public either.

There are also Atlas 800 training server with 8 Ascend 910 processors delivering and 32 GB HBM (1200 GB/s) 2.24 PFLOPS FP16 computing power; Atlas 900 PoD with 64 Ascend 910 processors and 2048 GB HBM delivering up to 17.92 PFLOPS FP16, supporting scaling to an AI cluster of up to 4096 Ascend 910 processors, delivering up to 1 EFLOPS FP16; and Atlas 900 AI Cluster consisting of thousands of Ascend 910 AI processors and the potential to deliver up to 256–1024 PFLOPS at FP16.

Atlas 900 AI Cluster

References:

DaVinci AI architecture

Both Ascend 910 and Ascend 310 contain DaVinci AI cores.

Two cores in Ascend 310 and 32 cores in Ascend 910.

There is MindSpore open source (Apache-2.0 License) deep learning training/inference framework from Huawei with native support for Ascend AI processors and software-hardware co-optimization.

There are also tools for Caffe/Tensorflow model conversion.

References:

Bitmain Sophon

Sophon (Beijing) is a subsidiary of Bitmain (a company developing crypto mining products), focusing on the development of artificial intelligence chips and artificial intelligence products.

Sophon is named after a proton-size supercomputer described in Cixin Liu’s “The Three-Body Problem” novel.

Sophon now has several generations of their Tensor Computing Processors: the BM1680 (1st generation, 2 TFLOPS FP32, 32MB SRAM, 25W), BM1682 (2nd gen, 3 TFLOPS FP32, 16MB SRAM), BM1684 (3rd gen, 2.2TFLOPS FP32, 17.6 TOPS INT8, 32 MB SRAM) and BM1880 (1 TOPS INT8).

Comparing to NVIDIA products such peak performance on FP32 is available with GTX 1050 Ti (~2 TFLOPS) or GTX 1060 (~3.4 TFLOPS).

Each BM1682 chip has 64 NPU processing units, and each NPU has 32 EU arithmetic units. BM1684 also has 64 NPUs.

SC5+

There are Deep Learning Acceleration PCIe Cards. The SC3 with a BM1682 chip (8 GB DDR memory, 65W), SC5 and SC5H with a BM1684 chip and 12 GB RAM (up to 16 GB) with 30W max power consumption, and SC5+ with 3x BM1684 and 36 GB memory (up to 48 GB) with 75W max power consumption.

Artificial Intelligence Server SA3 contains 6 or 18 BM1682, has 48 or 144 GB DDR memory, and can potentially deliver up to 18 or 54 TFLOPS.

For comparison, a single NVIDIA Titan RTX has an FP32 peak performance of 16.3 TFLOPS, and a single RTX 3090 has 35.6 TFLOPS FP32 (be careful, here we mention not tensor core performance, which is much higher, but it’s mixed-precision FP16/32).

BMNNSDK (BitMain Neural Network SDK) is BitMain’s proprietary deep learning SDK based on BM AI chip. BMNNSDK is composed of BMNet Compiler and BMRuntime.

BMNet Compiler is responsible for optimizing and converting various deep neural network models (Caffe, TensorFlow, Pytorch, Mxnet, PaddlePaddle, and Darknet (?)) finally converting it to the bmodel model supported by Bitmain TPU. BMRuntime is responsible for driving the TPU chip and providing a unified programmable interface for the upper-layer application program.

Alibaba Hanguang 800

Alibaba unveiled their first AI inference chip, Hanguang 800, during Apsara Conference on September 25, 2019.

The chip is optimized for Computer Vision Tasks, but GEMM is also accelerated.

A trained neural network model must be converted to INT8, and the software tools do support models from Caffe, MxNet, TensorFlow, and models in ONNX format.

It is SRAM only and has 192MB Local Memory and no DDR. Keep in mind, the models are compressed and quantized, so 192MB is actually not a very small number, but large models need to be split between several chips.

There are 4 cores with a ring bus. Each core has three engines: Tensor, Pooling, and Memory. The host interface is PCIe 4.0 x16.

The chip has a peak performance of 825 TOPS INT8. That’s really huge number for INT8! NVIDIA’s A100 INT8 performance is 624 TOPS, and twice that number for sparse calculations.

The chip has configurable frequency and voltage and can work in different modes optimizing for performance or lower power consumption, from 25W to 280W.

Interestingly, its performance is independent of the batch size.

Instances with Hanguang 800 are available in Alibaba Public Cloud.

Alibaba Group also has a RISC-V Xuantie910 processor with Vector Engine for AI Acceleration. It is rather relevant for our CPU section because it’s not ASIC, but a general-purpose processor. Its performance is more than 300 GFLOPS FP16 (32 FLOPS/core/cycle x 2.5 GHz x 4 Cores), and half that number for FP32. For comparison, the lowest consumer RTX series card, the NVIDIA RTX 2060 has 10000 GFLOPS for pure FP16 performance and even 51600 GFLOPS for FP16 on tensor cores, so that’s a different niche.

References:

Baidu Kunlun

July 2018, Baidu announced their AI chip called Kunlun. It is built on Baidu’s own XPU neural processor architecture.

The 14nm chip is produced at the Samsung Foundry. Kunlun offers 16GB HBM memory with 512 GB/s bandwidth and supplies up to 260 TOPS INT8 (that’s twice the INT8 performance of NVIDIA TESLA T4) or 64 TFLOPS INT16/FP16 at 150W. This chip looks like an inference chip.

The announcement of 2018 mentioned inference Kunlun chip 818–100 and training Kunlun chip 818–300. Their characteristics are unclear, but they are both mentioned among the supported hardware of Paddle Lite (a framework for inference, which seems to be analogous to TensorFlow Lite).

A Baidu Kunlun K200 board contains a single processor.

Baidu Kunlun software stack supports Paddle Paddle, Tensorflow, Pytorch with graph compiler. It also supports new operators by user-written kernels with XPU C/C++ programming language. Deep learning library has APIs for common operators used in a deep learning network.

The processor can be accessed via Baidu Cloud.

September 2020, Baidu announced Kunlun 2. The new chip uses 7 nm process technology and its computational capability is over three times that of the previous generation. The mass production of the chip is expected to begin in early 2021.

References:

Groq

Groq develops its own Tensor Streaming Processor (TSP). Jonathan Ross, Croq’s CEO had co-founded the first Google’s TPU before that.

Instead of creating a small programmable core and replicating it dozens or hundreds of times, the TSP houses a single enormous processor that has hundreds of functional units.

The 14nm chip contains 26.8B transistors. It has 220MB SRAM with 80TB/s on-die memory bandwidth and no board memory (it lacks DRAM controllers and interfaces) and supports PCIe Gen4 x16 with 31.5 GB/s in each direction (to download instructions and data to the TSP).

The chip delivers up to 1000 TOPS INT8 and 250 TFLOPS FP16 (with FP32 acc). For comparison, NVIDIA A100 has 312 TFLOPS on dense FP16 calculations with FP32 acc, and 624 TOPS INT8. The Groq’s number is even larger than the 825 TOPS INT8 of Alibaba’s Hanguang 800. Yet in the Linley Group report, Groq’s ResNet-50 inference performance in inferences/second is significantly lower than Hanguang’s one (20,400 vs 78,563).

The description of the architecture is taken from the Linley Group report.

Groq allows instructions executing at different times in different units. They flow first into a set of function units called Superlane 0, which executes these instructions. In the next cycle, they execute in Superlane 1, while Superlane 0 executes the second group of instructions. This technique simplifies the design and routing, eliminates the need for synchronization, and is easily scalable; the TSP chip features 20 superlanes. Within each superlane, data flows horizontally. In fact, the TSP continually pushes it across the chip on every clock cycle.

TSP conceptual diagram. Instructions flow downward through identical function units, pipelining the operations. Data flows across the processor, allowing the program to perform different operations.

Memory is embedded with the function units, providing a high-bandwidth data source and eliminating the need for external memory. The result is somewhat like a systolic array of heterogeneous function units, but the data only moves horizontally while the instructions move vertically.

The entire TSP executes a single instruction stream, so we consider it one processor core. But it’s effectively a 144-wide VLIW architecture, issuing one gross instruction per cycle to control the superlane. The TSP superlane is actually two sets of mirrored function units divided into what Groq calls the east hemisphere and the west hemisphere. Each function unit contains multiple subunits that accept instructions. For example, the vector unit contains 16 ALUs that are individually controlled.

A superlane comprises 16 lanes. Each instruction is performed on all 16 lanes at once, and then in the next superlane in the subsequent cycle, and so forth. Thus, over 20 cycles, each instruction executes on all 320 lanes across the 20 superlanes, so it effectively becomes a 320-byte SIMD operation having a 20-cycle pipeline.

TSP superlane block diagram. Every superlane is bilaterally symmetric with an east side and a west side. It contains 16 lanes, each of which is 8 bits wide. Data flows from east to west and from west to east.

From a programmer’s perspective, data is organized into streams, which physically comprise one byte per lane (320 bytes). The architecture supports 32 eastward streams and 32 westward streams. Each stream automatically progresses in its designated direction on every cycle, moving 32 bytes per lane. An instruction typically operates on data from different streams. For example, ADD S1, S2, S3 adds each value in stream 1 to the corresponding value in stream 2 and stores the results in stream 3. Thus, instead of a fixed set of 32 registers, each function unit operates on a moving set of 32 values.

The lane structure is optimized for INT8 data, but larger operands (INT16, INT32, FP16, or FP32) can be formed by combining streams. The superlane applies a 9-bit error-correction code (ECC) across all 16 lanes. The SRAM is also ECC protected.

The central vector unit contains 16 ALUs per lane. Each ALU can perform a 32-bit calculation using aligned groups of four stream bytes as operands. In addition to the usual arithmetic and logical operations, these ALUs can convert between integer and floating-point formats, perform the common activation functions (ReLU, tanh), and other math functions (exponentiation, reciprocal square roots).

The matrix units handle heavy computations. Each one contains 320 MAC units per lane that can be grouped into 20 supercells. On each cycle, MAC multiplies the stored weight values by a pair of activation values from the streaming data. Each 16x16 supercell can compute an integer partial sum in one cycle and a complete 320-element fused dot-product in 20 cycles. Each hemisphere has 320x320 MAC units producing 409,600 INT8 operations or 102,400 FP16 operations per cycle. Using all 32 streams in each direction, the TSP can load all 409,600 weight registers in less than 40 cycles.

The switch units can reshape tensor data to better suit the compute units (rotate, transpose, etc). The switch units also perform an important function as the only units that can communicate between superlanes.

Because the architecture lacks register files, the compiler must ensure the streaming data is available to the function unit at the designated time to execute the designated instruction.

The compiler must schedule all data movement, manage the memory and function units, and even manually fetch instructions. The Groq compiler orchestrates everything: Data flows into the chip and is plugged in at the right time and the right place to make sure calculations occur immediately, with no stalls. This allows Groq’s chip performance to be deterministic. The compiler reconfigures the hardware dynamically to perform each calculation so there is no abstraction between the compiler and the chip.

The software tools accept models developed in TensorFlow, and the company is developing drivers for other popular frameworks. Groq provides software that runs on an x86 host processor to download the neural network program (instructions) and data to the TSP; the accelerator can then autonomously execute the model and return results to the host.

The single-core Croq chip achieves peak throughput while processing one image at a time. This is similar to Hanguang 800 performance with no performance penalty for small batch sizes, the Groq architecture is hyper-focused on low latency, single-thread performance at batch size 1.

Image recognition inference performance for ResNet-50 v2 benchmarking at small batch sizes (source)

The Groq TSP accelerator is available on the Nimbix Cloud. It’s early access you need to apply.

References:

Qualcomm Cloud AI 100

The Qualcomm Cloud AI 100 processor is designed for AI inference acceleration in the cloud.

It’s a 7 nm processor with up to 16 cores. It has 144MB on-die SRAM (9MB for each AI Core) and up to 32GB LPDDR4x at 2.1GHz. The processor supports INT8, INT16, FP16, FP32 and for PCIe cards, it delivers up to 400 TOPS at 75W TDP. That’s a good number for just 75W…

Cloud AI 100 chip has a PCIe Gen3/4 x8 interface. It also has M.2 variants with lower TDP and performance.

The technical details are pretty scarce now.

The commercial launch is 1H 2021.

References:

Others

There are other players as well.

There were mentions of the Dataflow Processing Unit (DPU) by Wave Computing (the new owner of the MIPS architecture BTW). [1][2][3] But it’s unclear whether the company solutions for a datacenter or a desktop are available.

There are Arm solutions, but they seem to be for the edge. And, BTW, Arm is now NVIDIA.

There is SambaNova came out of stealth-mode in December 2020 with their Reconfigurable Dataflow Architecture (RDA) delivering “100s of TFLOPS”.

There is a company called Mythic that focuses on Compute-in-Memory, Dataflow Architecture, and Analog Computing.

We will be updating the article, adding new players here, and expanding them into separate sections.