Machine learning hardware (FPGAs, GPUs, CUDA) | Towards Data Science

There are options outside of GPUs (Graphics Processing Units) when it comes to deploying a neural network, namely the FPGA (Field Programmable Gate Array). Before delving into FPGAs and their implementation though, it’s good to understand a bit about GPU architecture and why GPUs are a staple for neural networks.
Popular libraries such as Tensorflow run using CUDA (Compute Unified Device Architecture) to process data on GPUs, harnessing their parallel computing power. This work is called GPGPU (General Purpose GPU) programming. It has been adapted to deep learning models which require at least thousands of arithmetic operations.
The deep convolutional neural network, as shown below, requires filters to be slid across pixel regions while outputting a weighted sum at each iteration. For each layer this process gets repeated thousands of times with varying filters of the same size. Logically, deep models get computationally heavy and GPUs come in handy.
Tensorflow can be built on the back of CUDA, which saves the end user from implementing parallel code and understanding the architecture of their chip. Its convenience and high optimization makes it perfect for widespread use.
FPGAs did not offer such a convenient solution earlier, using them required a deep understanding of how hardware works. But recent progress has made them more accessible and there’s more on that to come later.
Contents of this article assume little to no knowledge of how different hardware models function. It goes over the following:
GPUs, CPUs and CUDA
Advantages and design of FPGAs
HDL as a method for FPGA deployment
HLS as a method for FPGA deployment
FPGA deployment using LeFlow
Features of LeFlow for optimisation
CPUs (Central Processing Units) are designed for serial operations and supporting advanced logic. This is reflected in their design which contains less cores and more cache memory to quickly fetch complex instructions.
GPUs, however have hundreds of smaller cores for simple computation, and thus a higher throughput as compared to CPUs.
CUDA accesses a GPU’s many cores by abstracting them into blocks. Each block contains up to 512 accessible threads, with potentially 65 535 blocks able to run at once. Every thread executes a short program, and the catch is that it can run in parallel with other threads. Tensorflow takes advantage of this pattern to improve processing power, often running hundreds to thousands of threads simultaneously.
To learn more about using CUDA visit Nvidia’s Developer Blog or check out the book CUDA By Example.
Tensorflow is divided into two sections: library and runtime.
Library is the creation of a computational graph (neural network) and runtime is the execution of it on some hardware platform.
The preferred platform is a GPU, however there is an alternative: FPGAs.
FPGAs can produce circuits with thousands of memory units for computation, so they work similarly to GPUs and their threads in CUDA. FPGAs have adaptable architecture, enabling additional optimisations for an increase in throughput. Thus the possible volume of calculations makes FPGAs a viable solution to GPUs.
Comparatively FPGAs have lower power consumption and can be optimal for embedded applications. They are also an accepted standard in safety-critical operations such as ADAS (Advanced Driver Assistance Systems) in automotive.
Furthermore, FPGAs can implement custom data types whereas GPUs are limited by architecture. With neural networks transforming in many ways and reaching out to more industries, it is useful to have the adapatability FPGAs offer.
An FPGA (Field Programmable Gate Array) is a customisable hardware device. It can be thought of as a sea of floating logic gates. A designer comes along and writes down a program using a hardware description language (HDL), such as Verilog or VHDL. That program dictates what connections are made and how they are implemented using digital components. Another word for HDL is RTL (register-transfer level) language.
FPGAs are easy to spot, look for an oversized Arduino.
Just kidding, they come in all shapes and sizes.
Using software analogous to a compiler, HDL is synthesized (figure out what gates to use), then routed (connect parts together) to form an optimized digital circuit. These tools (HDL, synthesis, routing, timing analysis, testing) are all encompassed in a software suite, some include Xilinx Design Tools and Quartus Prime.
Currently models get trained using a GPU, but then are deployed on an FPGA for real-time processing.
For FPGAs, the tricky part is implementing ML frameworks which are written in higher level languages such as Python. HDL isn’t inherently a programming platform, it is code written to define hardware components such as registers and counters. Some HDL languages include: Verilog, VHDL.
Shown below is a snippet of some code used to create a serial bit detector.
If you’re unfamiliar with it, try guessing what it does.
Done? Even if you stare at it for a while, it isn’t obvious.
Most of the time FSMs (Finite State Machines) are used to split the task up into states with input-dependent transitions. All of this is done before programming, to figure out how the circuit will work per each clock cycle. Then this diagram, as shown below, gets converted into blocks of HDL code.
Back to the topic: main point is that there’s no direct translation to convert a loop in Python to a bunch of wires in Verilog.
Given the possible complexity of a design, it can be very difficult to debug it for further optimisation. There are no abstractions to simplify the process as there would be in CUDA, where a thread can be selected and modified.
Well no, FPGAs aren’t useless.
One way to work around the programming problem is to use HLS (high level synthesis) tools such as LegUp to generate programs in Verilog for deployment. HLS tools allow designers to avoid writing HDL from scratch and instead use a more intuitive, algorithmic programming language (C).
HLS tools abstract away hardware-level design; similar to how CUDA automatically sets up concurrent blocks and threads when the model is run.
HLS tools require C code as an input which gets mapped to an LLVM IR (intermediate representation) for execution. The tools are used to convert procedural descriptions to a hardware implementation.
Their role in FPGA design is shown below.
LLVM is not an acronym, it is a library that constructs assembly-like instructions (IRs). These programs are easier for HLS tools to process and can be used to create synthesizable code for an FPGA.
IRs are used to describe source code in a general format, allowing use by various programs.
To learn more about LLVM and IRs, refer to Dr. Chisnall’s slides.
The main issue is converting programs and transferring libraries written for Python to C for HLS tools to function. Currently there is no support for Tensorflow in C, so this solution is very difficult. Evidently the requirement for laying out and creating hardware is a large barrier to the use of FPGAs in deep learning.
The LeFlow Toolkit allows engineers to design, train and test their models using Python, then deploy it directly to an FPGA for use. LeFlow simplifies the design process by allowing HLS tools to be compatible with Python and Tensorflow, acting as an adapter.
The software was designed by researchers within the ECE (Electrical and Computer Engineering) department at the University of British Columbia: Daniel H. Noronha, Bahar Salehpour, and Steven J.E. Wilson.
The following sections detail how LeFlow is integrated with Tensorflow and an FPGA, if you’re only interested in implementation skip to: Tuning Time.
An XLA (Accelerated Linear Algebra) compiler made for Tensorflow outputs an LLVM IR. LeFlow restructures the IR and optimizes it to be used with HLS tools.
From thereon, HLS tools do all the leg work to convert that IR to a program deployed onto FPGAs as described in the section: Should We Stick to GPUs?
LeFlow takes the IR as an input. Algorithm 1 is an IR of Tensorflow loading two floats.
The program is difficult to follow and looks messy. LeFlow will clean it up and change it.
Its goal is to create global variables, then map them as inputs and outputs of a hardware interface. The diagram shown below contains a general overview of the synthesized circuit after LeFlow re-formats and passes the IR through LegUp.
Evidently a hardware interface requires additional modules and signals such as clock, reset, memory, and memory controller. LegUp handles the creation of these parts including timing specifications for the clock.
LeFlow sets up a volatile load for FPGA registers. This allows variable access, and automatic circuit modification when high-level code is changed. Changes are evident in lines 1, 6, and 7 mainly.
LeFlow finishes its job as shown by the revamped IR of Algorithm 2, and the rest is handled by an HLS tool!
An interesting feature of LeFlow is the ability to change hardware. LeFlow offers unrolling, and memory partitioning parameters which turbocharge computations when used correctly. This combined with the low latency inherent to FPGAs allows them to work with exceptional efficiency.
The great thing is that these parameters can be specified in Python and the instructions get passed on directly down to the circuit level.
Unrolling is used for looping and is a careful balancing act. The idea is to do multiple calculations (or replications) per iteration and take larger steps.
This looks the same as a regular for loop, but there are more components added at the hardware level to perform more calculations per clock cycle (or loop iteration). More on unrolling is available at Keil’s User Guide.
int sum = 0;for(int i = 0; i < 10; i+=2)     sum += a[i]; sum += a[i+1];
Looking above we can unroll by a factor of two, meaning two iterations are taken at once and the loop increases by two steps.
This takes advantage of an FPGA’s parallelism to work more like a GPU, evidently per this example cycles were reduced by 13%.
Keep in mind that additionally added hardware can cause inefficiency or size constraints.
The HLS used in LeFlow’s pipeline requires a dual-port RAM (Random Access Memory) for storing values.
This is problematic as RAM is designed to store lots of data, but at a cost of being very slow. It can take ten times the amount of clock cycles to fetch values from it.
Luckily FPGAs contain many other independent memory units, so LeFlow can partition its data into multiple unique storage points. In a sense its analogous to adding more cores in a processor. It reduces clock cycles by allowing more instructions to execute concurrently.
Imagine the task is to multiply elements of two arrays of size eight together. In parallel computing, tasks get split up into groups to be executed simultaneously. A cyclic decomposition means a certain set of steps get repeated at the same time.
FPGAs can run more efficiently using memory partitions. The following schedule has been run over eight clock cycles.
In (a) there are no memory partitions, so one element from each array gets loaded, multiplied, and stored per cycle. The process continues and take eight cycles till completion.
In (b), arrays are cyclically partitioned into two separate memories. Two elements from each array are loaded, multiplied, and stored per cycle. Larger chunks indicate that processes occur simultaneously albeit in different sections of hardware. This reduces it down to six cycles.
In c), arrays are cyclically partitioned into four separate memories and there is a reduction to five cycles. Four elements from each array are loaded, multiplied, and stored per cycle.
Once LeFlow is set up correctly, all it needs to run without any additional configurations (such as unrolling) is a device selection line:
with tf.device("device:XLA_CPU:0")
It indicates the XLA-compiler will be used to generate LLVM and start the LeFlow conversion process.
Now you’re an expert on understanding how LeFlow works and maybe FPGAs are right for you.
There are loads of examples of LeFlow and its specific installation on Github. Daniel Holanda (one of the co-authors) has source code up for MNIST digit recognition among other things, so grab an FPGA and give it a go!