arXiv:1912.03413v1 [cs.DC] 7 Dec 2019

Dissecting theGraphcore IPU
Architecture
via Microbenchmarking
Technical Report
December 7, 2019

Zhe Jia
Blake Tillman
Marco Maggioni
Daniele P. Scarpazza

High Performance Computing R&D Team
Citadel, 131 S. Dearborn St., Chicago.

Copyright c 2019, Citadel Enterprise Americas, LLC. All rights reserved.
Chicago, United States of America.
Edition

Date

First

December 7, 2019

The authors make this document available on the arXiv.org e-print service
owned and operated by Cornell University. Citadel grants arXiv.org a perpetual, non-exclusive license to distribute this manuscript.
Disclaimer. This presentation solely reflects the analysis and views of the authors. No recipient should interpret this presentation to represent the generalviews of Citadel or its personnel. Facts, analyses, and views presented herein
have not been reviewed by, and may not reflect information known to other
Citadel professionals.
Objectivity. This research aims at being objective and impartial. The authors
strive to ensure that each claim is based on solid evidence. Results publishedhere are meant to be reproducible by anybody who recreates the experimental
conditions described by the authors, with the possible exception of good faith
mistakes. No such mistakes are known to the authors at the time of publication.
Conflicts of interest. This research was supported entirely by Citadel. Citadel
received no financial support from Graphcore or any other entity in connection with this work. Graphcore helped this work by providing an earlyavailability test system to Citadel, and by reviewing the contents of thismanuscript for mistakes and omissions. The authors did not receive any
consulting fees, research funding or reimbursement in conjunction with thiswork, and are not employed by Graphcore.
Acknowledgments. The authors thank their colleague Jeffrey Smith for his
valuable input and assistance during the preparation of this manuscript.
All product names, trademarks and registered trademarks are property of
their respective owners.

Contents
Contents

3

1

Architecture
1.1 Design Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Fine-grained Parallelism . . . . . . . . . . . . . . . . . . . . . . .
1.3 Arithmetic Throughput . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Memory Architecture . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Interconnect Architecture . . . . . . . . . . . . . . . . . . . . . .
1.6 The Bulk Synchronous Parallel Model . . . . . . . . . . . . . . .

7
7
9
11
12
13
15

2

Experimental Setup
2.1 System Configuration . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Multi-IPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17
17
21
23

3

Local Memory
3.1 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25
25
26

4

Interconnect
4.1 Point-to-point Transfers . . . . . . . . . . . . . . . . . . . . . . .
4.1.1 Congestion-free Latency . . . . . . . . . . . . . . . . . . .
4.1.2 Latency between Tiles on a Chip by Proximity . . . . . .
4.1.3 Latency between IPUs by Proximity . . . . . . . . . . . .
4.1.4 Latency under Load . . . . . . . . . . . . . . . . . . . . .
4.1.5 Congestion-free Peak Bandwidth . . . . . . . . . . . . . .
4.1.6 Peak Bandwidth under Load . . . . . . . . . . . . . . . .
4.1.7 Peak Bandwidth between IPUs by Proximity . . . . . . .
4.1.8 Multi-threaded Peak Bandwidth . . . . . . . . . . . . . .
4.2 Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Congestion-free Broadcast Latency . . . . . . . . . . . . .
4.2.2 Peak Broadcast Bandwidth . . . . . . . . . . . . . . . . .
4.2.3 Effect of Message Size on Broadcast Bandwidth . . . . .
4.3 Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Congestion-free Gather Latency . . . . . . . . . . . . . .

31
31
32
33
34
37
39
41
43
44
46
47
50
51
54
54

3

4

CONTENTS
4.3.2 Peak Gather Bandwidth . . . . . . . . . . . . . . . . . . .
Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Congestion-free Scatter Latency . . . . . . . . . . . . . .
4.4.2 Peak Scatter Bandwidth . . . . . . . . . . . . . . . . . . .
All to all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6.1 Minimum Reduction Latency - Weak Scaling . . . . . . .
4.6.2 Minimum Reduction Latency - Strong Scaling . . . . . .
4.6.3 Peak Reduction Bandwidth . . . . . . . . . . . . . . . . .
Host Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7.1 Minimum Host-to-IPU Latency . . . . . . . . . . . . . . .
4.7.2 Peak Host-to-IPU Bandwidth . . . . . . . . . . . . . . . .

56
57
58
59
60
60
61
64
64
67
67
67

Notable Arithmetic Primitives
5.1 Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Pseudo-Random Number Generation . . . . . . . . . . . . . . .

71
71
75
83

4.4

4.5
4.6

4.7

5

List of Figures

86

Bibliography89

Abstract
This report focuses on the architecture and performance of the Intelligence
Processing Unit (IPU), a novel, massively parallel platform recently introduced by Graphcore and aimed at Artificial Intelligence/Machine Learning(AI/ML) workloads.
We dissect the IPU’s performance behavior using microbenchmarks that
we crafted for the purpose. We study the IPU’s memory organization andperformance. We study the latency and bandwidth that the on-chip and offchip interconnects offer, both in point-to-point transfers and in a spectrum ofcollective operations, under diverse loads. We evaluate the IPU’s compute
power over matrix multiplication, convolution, and AI/ML primitives. We
discuss actual performance in comparison with its theoretical limits.
Our findings reveal how the IPU’s architectural design affects its performance. Moreover, they offer simple mental models to predict an application’s
performance on the IPU, on the basis of the computation and communication
steps it involves.
This report is the natural extension to a novel architecture of a continuing
effort of ours [1, 2] that focuses on the microbenchmark-based discovery ofmassively parallel architectures.

5Chapter 1

Architecture
In this chapter, we introduce the reader to the fundamentals of the IPU’s architecture and its programming paradigm.

1.1

Design Philosophy

The IPU architecture and its compute paradigm were co-designed from theground up specifically to tackle machine intelligence workloads. For thatreason, they incarnate certain design choices that depart radically from morecommon architectures like CPUs and GPUs, and might be less familiar to thereader. This section and the following ones discuss these choices and how
they affect applications and application designers.
At the heart of any IPU-based system is the IPU processor, of which weoffer a simplified block diagram in Figure 1.1. Its design aim is the efficient
execution of fine-grained operations across a relatively large number of parallel threads. This means that the IPU, unlike other massively parallel architectures (e.g., the GPU) adapts well to fine-grained, irregular computation thatexhibits irregular data accesses. The IPU offers true MIMD (Multiple Instruction, Multiple Data) parallelism and has distributed, local memory as its only
form of memory on the device.
Each IPU contains 1,216 processing elements called tiles; a tile consists of
one computing core plus 256 KiB of local memory. Except for the register file,the IPU offers no memory other than the distributed memories local to eachtile.
In addition to the tiles, the IPU processor contains the exchange, an onchip interconnect that allows for high-bandwidth, low-latency communication among tiles.
Each IPU also contains ten IPU link interfaces; the IPU link is a Graphcoreproprietary interconnect that enables low latency, high-throughput commu7

8

CHAPTER 1. ARCHITECTURE
Tile
Local
memory

Core

IPU
Link

IPU
Link

IPU
Link

IPU
Link

PCI
Link

IPU
Link

IPU Exchange

PCI
Link

IPU
Link

IPU
Link

IPU
Link

IPU
Link

IPU
Link

Figure 1.1: Simplified block diagram of an IPU processor: the processor features 1,216 tiles(each containing one core and its local memory), the exchange (an on-chip interconnect), IPU linkinterfaces that provide connectivity to other IPU chips, and PCIe interfaces for host connectivity.

1.2. FINE-GRAINED PARALLELISM

9nication between IPU processors. Because IPU links make transfers between
remote tiles as simple to the programmer as between local tiles, they are thelinchpin to the IPU paradigm’s scalability. Finally, the IPU contains two PCIe
links for communication with CPU-based hosts.
Terminology. While Graphcore’s commercial literature uses the terms IPU-TilesTM ,IPU-CoreTM , IPU-ExchangeTM and IPU-LinksTM , we refer to the same components astiles, core, exchange and IPU links, respectively, with no risk of confusion.
Source. Information we report here on IPU architecture derives from Graphcore’s technical literature or from direct correspondence with Graphcore, and is republished withpermission.

1.2

Fine-grained Parallelism

The IPU’s emphasis on fine-grained parallelism means that the IPU can efficiently run applications that have irregular and sparse data access patterns
and control flow. Unlike SIMD/SIMT architectures, IPUs don’t need large
warps of threads consuming contiguous vector data to achieve high efficiency.
Instead, IPUs can run individual processing threads on smaller data blocks,
in a highly parallel MIMD fashion. Each thread can have completely distinct
code and execution flow without incurring performance penalties.
Architecturally, IPUs differ significantly from platforms commonly used
to execute ML/AI workloads, namely CPUs and GPUs. We discuss thosedifferences by comparing the fundamental approaches behind the design ofCPUs and GPUs with those behind IPUs.
CPUs
CPUs tend to offer complex cores in relatively small counts. CPU cores feature sophisticated latency-reducing techniques like branch prediction, branch
speculation, and out-of-order execution. These optimizations make CPUs excel at single-thread performance and control-dominated code, possibly at theexpense of energy efficiency and aggregate arithmetic throughput per area ofsilicon.
Even if most modern CPUs offer vectorization (SIMD, Single Instruction,
Multiple Data), they can’t match GPUs in aggregate floating-point arithmetic
or in energy efficiency (performance per Watt) on large, regular, array-based
workloads.
To hide memory latency, CPUs typically employ a deep memory hierarchy
containing multiple levels of caches, together with prefetching and sophisticated prefetch predictors.
GPUs

10

CHAPTER 1. ARCHITECTURE

GPUs, in contrast, feature smaller cores in a significantly higher count per
device (e.g., thousands). GPU cores are architecturally simpler than those ofCPUs and do not typically offer branch speculation, sophisticated branch prediction, out-of-order execution, or hardware prefetching.
GPUs arrange their cores into clusters that operate in lockstep; all cores in acluster execute the same instruction at any point in time. Threads are grouped
together in warps; warps are scheduled together to clusters. Threads in a warp
will perform the same operation on independent data. This execution modelis referred to as SIMT: Single Instruction, Multiple Threads.
Even though GPUs have evolved deep memory hierarchies featuring bothcache and scratchpad memories [1, 2], their fundamental approach to hiding
memory latency remains the oversubscription of threads to cores and the ability to inexpensively switch among threads. In this approach, when a warp ofthreads is awaiting operands from main memory, the hardware can suspendthem and switch to another warp that has received its operands from memory
and is ready to continue. The programming model explicitly encourages developers to expose sufficient thread parallelism so that a portion of the threads
are always ready to execute, while the remainder await operands.
GPUs only access main memory at peak throughput when load/store operations within each warp involve contiguous regions of memory; that accessis said to be coalesced. Uncoalesced memory accesses only achieve a fractionof the theoretical peak memory bandwidth. For best performance, developers
must instantiate large blocks of threads that share the same program, attempt
to consume input data (and produce output) in a coordinated, coalesced manner, and use control flow sparingly. Because of the SIMT model, GPUs pay a
performance penalty when threads diverge in their control flow.
Because of their organization, GPUs excel at regular, dense, numerical,
data-flow-dominated workloads that naturally lead to coalesced accesses anda coherent control flow. On these workloads, GPUs also tend to be more energy efficient than CPUs because they dedicate a higher fraction of their silicon
to arithmetic units, rather than caches and latency-oriented features. Moreover, GPUs refactor instruction decoding circuitry outside of individual cores;
this is possible because clusters of cores operate on the same instruction.
IPUs
IPUs provide large core counts (1,216 per processor) and offer cores complex enough to be capable of executing completely distinct programs. The
IPU’s approach to reducing memory latency is radical—it does away withshared memory entirely. The IPU only offers small, distributed memories thatare local and tightly coupled to each core. Each tile contains 256 KiB of memory, totaling 304 MiB per device.
IPU memories are scratchpads, not caches. They are implemented as1.3. ARITHMETIC THROUGHPUT

11

SRAM and therefore offer much higher bandwidth (45 TB/s, aggregate) andlower latency (6 clock cycles) than DRAMs. Their performance is comparable to L2 CPU caches and superior to GPU shared memories (or L1 caches).
Section 1.4 extends our discussion on memory.
IPU cores pay no penalty when their control flows diverge or when theaddresses of their memory accesses diverge. In fact, they pay no penalty forrunning disjoint instruction flows that exhibit uncorrelated memory accesses.
Cores access data from their respective local memory at a fixed cost that is
independent of access patterns. This makes IPUs more efficient than GPUs at
executing applications with irregular or random data access patterns and/or
applications that are control-flow dominated, provided that working sets fit
in IPU memory.
Similarly to CPUs and GPUs, IPUs achieve higher efficiency by oversubscribing threads to cores. Specifically, each IPU tile offers hardware support
for 6 threads in a manner that is functionally similar to the SMT technique (Simultaneous MultiThreading, or Intel’s Hyper-Threading) commonly foundon CPUs. Each IPU tile maintains 6 resident execution contexts and multiplexes them onto shared resources, thus hiding instruction latencies (dependency, memory access and branch latencies), reducing corresponding pipeline
stalls, and increasing aggregate throughput. Each tile rotates among threads
according to a static, round-robin schedule. The entire IPU supports therefore
6 × 1,216 = 7,296 threads. For maximum occupancy, software designers areencouraged to instantiate that many threads.
On the IPU, efficiency on irregular workloads does not come at the expenses of regular, numerical, array- or matrix-based workloads, which are
known to run well on GPUs. In the next section we show that IPUs outperform GPUs on a per-board comparison when operands fit in memory.

1.3

Arithmetic Throughput

The IPU offers an impressive arithmetic throughput, up to 31.1 TFlops/s insingle precision and 124.5 TFlops/s in mixed precision1 per chip, surpassing
contemporary GPUs in a comparison of theoretical limits.
In a per-board comparison, the IPU’s theoretical advantage over the GPU
grows roughly by a factor of two, and so it does in an energy efficiency comparison.
This level of throughput is made possible by the use of specialized
pipelines called Accumulating Matrix Product (AMP) units that are present ineach IPU tile. AMP unit are used to accelerate matrix multiplication and con1 With mixed precision we denote operations in which multiplicands are in half precision (FP16)
and their products are accumulated onto a single precision result (FP32).

12

CHAPTER 1. ARCHITECTURE

volution operations. An AMP unit can finish 64 mixed-precision or 16 singleprecision floating point operations per clock cycle.
However, actual arithmetic performance, both on IPUs and GPU, depends
dramatically on the properties of the specific numerical workload at hand,and may significantly differ from theoretical limits. Evaluating the IPU’s arithmetic performance on a meaningful basket of real-world numerical workloads
is far outside of the scope of a microbenchmarking report like this. However,
we benchmark matrix multiplication performance as offered by Poplar’s linear algebra library (poplin) and we compare them with theoretical limits, andwith respective performance numbers associated with NVidia’s V100 GPU.
The actual performance we measured shows the IPU as a clear winner insingle precision against NVidia’s V100 GPU (per-chip comparison). In mixed
precision, the comparison does not yield a clear winner and requires a morenuanced discussion. We postpone this discussion, together with experimental
setup details and quantitative results to Section 5.1, which is entirely dedicated to the matrix multiplication workload.

1.4

Memory Architecture

In the IPU’s fine-grained processing philosophy, the role of local memories isfundamental. In fact, the very choice to adopt distributed SRAM memories
located next to the cores is what allows threads to access data efficiently evenwhen access patterns are irregular, sparse and incoherent at a fine grain.
Each tile contains 256 KiB, totaling 304 MiB on the entire processor. Eachtile owns an independent, contiguous 21-bit address space that is shared
among the 6 hardware execution contexts, where code executed locally anddata processed locally must fit. The nominal aggregate bandwidth for theentire IPU memory is 45 TB/s, while the latency is 6 clock cycles.
While the IPU’s aggregate capacity is lower than the typical DRAM memory on a GPU (e.g., 32 GiB), IPU memories make up in speed what they lackin capacity; they have shorter latency than both L1 caches and shared memories on the NVidia Turing T4 GPUs, and comparable latency with L2 caches
on Intel Skylake/Kaby Lake/Coffee Lake CPUs. See Table 1.1 for a quantitative comparison between IPU memory and SRAM memories with comparable
latency on GPUs and CPUs.
As far as capacity is concerned, the aggregate IPU memory is larger thanmemory layers of equivalent latency on CPUs and GPUs, surpassing them by
one to two orders of magnitude (see column Per-chip Capacity in Table 1.1). The
aggregate size of the IPU’s memory removes the need for a cache hierarchy
similar to those found on GPUs and CPUs.
We dedicate the entirety of Chapter 3 to studying the performance of the
IPU’s local memory, which we measure via microbenchmarks.

1.5. INTERCONNECT ARCHITECTURE

13

Architecture

MemoryPer-chipCapacity(MiB)

Latency(ns)

Latency(cycles)

Clock
Frequency(GHz)

Graphcore IPU

Tile-local

304

3.75

6

1.60

NVidia T4 GPU

SharedL1

1.25 ... 2.5
1.25 ... 2.5

11.94
20.13

19
32

1.59

Intel *-Lake CPU

L1D
L2

0.25 ... 0.875
4 ... 280.93 ... 1.92
2.79 ... 4.62

4 ... 512

2.60 ... 4.30

Table 1.1: Size and latency comparison between IPU memories and similar SRAM-based memory hierarchy levels on contemporary GPUs and CPUs. The IPU’s local memory has lower latency
than the fastest memories on the Turing GPUs and is on par with the L2 cache in modern Intel
CPUs. However, the IPU’s local memory is vastly larger than those memories in a per-chip comparison. (Intel data: from public sources; intervals range over the product offering at the time of
the writing. GPU data: from our prior work [2].)The IPU’s memory organization requires software designers to partition
their working set across the tiles’ memories appropriately and make tiles exchange data with each other when they need non-local operands. The programming model, along with Graphcore’s Poplar language and associatedcompiler, allows for the automatic orchestration of these data transfers. Designers describe the operands’ flow but need not worry about explicit variable
placement and transfer scheduling. Transient data that is only consumed once
should be streamed into the device via PCI from the host.
Graphcore’s optimized machine-learning and linear algebra libraries
adopt the data partitioning approach we described. Performance-sensitive
software designers are encouraged to follow it as well.
The cumulative memory of 304 MiB per IPU (608 MiB per board) is typically sufficient for many models used in contemporary AI/ML applications toreside entirely on-chip. Models that fit entirely on chip benefit from the highbandwidth and low latency offered by local memory. Models of larger size canbe sharded across IPU processors and IPU boards, thanks to the architecturaland programming paradigm features described in the next section.

1.5

Interconnect Architecture

The IPU interconnect is what allows tiles on an IPU system to work tightly
together and exchange data efficiently with each other. It is also what truly
makes a system with multiple IPUs act as a single, coherent machine.
A system with multiple IPUs exposes the single IPU devices independently, but it also exposes Multi-IPUs. A Multi-IPU is a virtual IPU device
that is comprised of multiple physical IPUs and offers all their memory andcompute resources as if they belonged to a single device. The ability to federate multiple physical IPUs into a virtual, monolithic device is precisely what

14

CHAPTER 1. ARCHITECTURE

allows users to train and infer models larger than a single IPU’s capacity, while
also taking advantage of the cumulative compute power. Software designers
can scale their applications to multiple IPUs with no additional development
effort because the same APIs can target physical IPUs or Multi-IPUs indifferently. The tight cooperation between on-chip exchanges and IPU links is the
crucial factor that allows Multi-IPUs to exist and put the cumulative memory
capacity and compute resources of its participants at the developer’s disposal.
To study the IPU’s on-chip and off-chip interconnect performance, we
adopt the terms, the methods and the interests of classic research focusing on the characterization of parallel systems and high-performance networks [3, 4, 5, 6].
On chip, the interconnect exhibits an impressive aggregate throughput
of 7.7 TB/s (actual). We measured this throughput under load, with alltiles transferring data concurrently to randomly selected destinations, witha benchmark that is representative of all-to-all exchanges (Section 4.1.6). On a
per-tile basis, each of the 1,216 tiles can simultaneously use 6.3 GB/s of bandwidth to transfer data to an arbitrary destination on chip. The latency of an
on-chip tile-to-tile exchange is 165 nanoseconds or lower and does not degrade under load above that value.
In multiprocessor systems, the exchange and the IPU links work together
to support tile-to-tile communication transparently to the user, regardless of
where in the system the two endpoints are located; it is as easy to program a
Multi-IPU as it is a single, physical IPU.
Each board is connected to peer boards via IPU links and to its host system via PCIe interfaces. The 2 IPUs on each board are connected by threelinks with a nominal bidirectional bandwidth of 64 GB/s each, two of which
reserved for intra-board transfers. Our benchmarks achieved an actual bandwidth of 108 GB/s.
Connecting boards in an IPU-link network is the key to building larger
IPU-based systems. While our study is limited to a single-host configuration
featuring 8 boards, much larger systems can be built, including hostless systems. They are beyond the scope of this report. Although different network
topologies are possible, our experimental evaluation focuses on the concreteconfiguration adopted in the test system provided to us by Graphcore; we
describe it in detail in Chapter 2.
Then, we dedicate the entirety of Chapter 4 to the study of the IPU’s interconnect performance, both on chip and across chips.
In summary, we remark two general observations:• performance: the aggregate arithmetic resources of the virtual IPU scalelinearly, and the overall interconnect performance scales relatively well.

1.6. THE BULK SYNCHRONOUS PARALLEL MODEL15

For example, communication latencies degrade gracefully with systemdiameter, as our benchmarking results show in Sections 4.1.1 and 4.1.3;• programmability: the Multi-IPU programming model is transparent to
the developer. The underlying hardware makes the abstraction efficientand, in practice, no extra development effort is needed to scale applications onto large IPU systems. In contrast, CUDA applications do require
extra development effort and complexity to parallelize across multiple
GPUs, and even more to parallelize across hosts. The same is true for
CPU parallel applications, especially across hosts.

1.6

The Bulk Synchronous Parallel Model

An important design factor that underlies the IPU programming paradigm isthe Bulk Synchronous Parallel (BSP) model [7]. BSP is the very approach that
IPUs use to organize their compute and exchange operations.
Proposed in the 1980s, the BSP model is an abstraction for parallel computation that facilitates expressing parallel algorithms and reasoning on theperformance they achieve as they execute on parallel computers.
The BSP model organizes computation in multiple sequential supersteps; asuperstep is composed of a local computation phase, followed by a communication phase and a barrier synchronization:• in the local computation phase, every process performs computationthat operates solely on local memory. No communication between processes occurs in this phase;• in the communication phase, processes exchange data; each process
may send a message to each desired destination counterpart (all-to-all
personalized exchange). No computation occurs in this phase;• a barrier synchronization phase follows; no process continues to thenext superstep till all processes have reached the barrier. Neither computation nor communication occurs in this phase except for that strictly
required by the barrier itself.
Processes can use the communication phase not only to send each otherintermediate computation results, but also to request and (at a later communications stage) receive data from remote memories. This mechanism allows
each process to use any other’s local memory as a remote memory and to ultimately access the entire aggregate system memory as one larger store.
Parallel algorithms of arbitrary complexity can be described in the BSP
model without restriction of generality.

16

CHAPTER 1. ARCHITECTURE

The IPU is a true BSP machine. It faithfully incarnates hardware support,
enforcement and optimization for the three phases of each BSP superstep. Its
programming model lets programmers specify processes in terms of graphvertices that compute on local data (and local data only).
Input operands are brought to each process by the run-time system before the computation begins, in the communication step associated with theprevious superstep.
The programming model and the hardware jointly enforce the separation
between phases:• IPU cores can only access directly local memories; this organization naturally enforces the local restriction of the computation phase;• the on-chip exchange provides native hardware support and acceleration for the all-to-all exchanges of the communication phase ...
• and for the barrier synchronization as well.
In the IPU paradigm supported by the Poplar SDK, programmers describe
computation as vertices, data as tensors and data exchanges as static edges,
without having to worry about allocation of data at rest in local memories, allocation of input/output transfer buffers, or scheduling of transfers. All thesetasks are carried out by the Poplar compiler. The compiler organizes the processing of machine intelligence models exclusively using this BSP paradigm.
Because the IPU implements the BSP model using its exchange and IPU
links, the interconnect’s performance is the primary factor that determines
the performance of BSP communication and synchronization phases and ultimately affects overall application performance. We dedicate Chapter 4
to study the performance of intra- and inter-IPU transfers via benchmarks.
Those benchmarks are, in fact, compiled into and executed as BSP communication and barrier synchronization phases.

Chapter 2

Experimental Setup
In this chapter, we specify the experimental system setup we use in our benchmarks. Of particular interest to the reader is the IPU link network topology,
which determines the IPUs’ relative proximity to each other and influences
the performance of data transfers across IPUs.

2.1

System Configuration

All results in this report derive from benchmarks that we ran on a test systemprovided by Graphcore and equipped with 8 Graphcore C2 PCIe boards. EachC2 board hosts 2 IPU processors running at 1.6 GHz. While this test system
was configured for 1.6 GHz IPU operation, production systems may differ in
configuration and performance; contact Graphcore directly for inquiries.
We show a simplified representation of the server’s topology and its connections in Figure 2.1. The system features two Intel Xeon Platinum 8168
CPUs, each containing 24 cores, with 33 MiB of L3 cache and a clock frequencyof 2.70 GHz.
IPUs are connected to each other via IPU links, organized in a ladder network (thin and thick blue arrows in the figure). The network topology directly
explains the interconnect performance that we measured (Chapter 4).
In this ladder-shaped topology,• each vertical “side rail” represents one single-link chain connecting either all even or all odd IPUs; each such link has a nominal 64 GB/s
bandwidth;• each horizontal “rung” (thick, horizontal, blue arrow) represents a
triple-link, bundled connection between the two IPUs on each C2 board.
The three links offer a nominal bidirectional bandwidth of 64 GB/s each,or 192 GB/s aggregate.
17

18

CHAPTER 2. EXPERIMENTAL SETUP

Aft

Fwd

LegendC2 board

x16 IPU-Link 64 GB/s

IPU7

IPU5

3x x16 IPU-Link 192 GB/s
x16 PCIe Gen3 32 GB/s

C2 board
IPU6

IPU4

IPU1

IPU3

IPU0

IPU2

C2 board

C2 board

PCIe
FabricC2 board
IPU15

Host
CPUs
and
memoryIPU13

C2 board
IPU14

IPU12

C2 board
IPU11

IPU9

C2 board
IPU8

Aft

IPU10

Fwd

Figure 2.1: Topology and interconnection of C2 boards and IPU processors in the system that we
employed in all our experiments. All IPU numbers in this figure are Device IDs. Board placementin the chassis is depicted accurately (see Fwd and Aft arrows).

2.1. SYSTEM CONFIGURATION19

The routing of traffic along the bundled on-board IPU links is staticallyconfigured so that:• two links are reserved for data transfers where the two IPUs on a board
are the source and destination; this yields a 128 GB/s nominal aggregate
bidirectional bandwidth between the two IPUs of each board;• one link is reserved for pass-through transfers, where either the source or
the destination IPU belongs to another board and has a different parity.
In the topology, each board has exactly two neighbors, with the exceptionof the first and last boards, which only have one neighbor. Each IPU is connected directly to the other IPU on the same C2 board and up to two other
IPUs located on neighboring boards. For example, IPU1 is connected directly
to IPU3 (on the same board), and to IPU0 and IPU6 (on different boards). IPU0
is not directly connected to IPU3; each data transfer from IPU0 to IPU3 uses
either IPU1 or IPU2 as a relay.
For the avoidance of doubt, the ladder structure is not a torus, i.e., it does
not wrap around at the edges. The card hosting IPUs 7 and 5 is not a neighborof the card hosting IPUs 8 and 10. A transfer from IPU7 to IPU8 must traverse
the entire ladder vertically.
All the IPU numbers we use here are Device ID numbers as exposed andused by the Poplar SDK and by the IPU’s command-line tools. Device IDs reflect the lexicographic order of the respective devices’ PCI domains, not their
physical placement in the server. For these reasons, Device IDs do not respectproximity in the network topology. IPUs with consecutive Device IDs are noton the same board or, in general, neighbors. For example, consider IPU pair
7 and 8. On the other hand, Poplar provides a mapping (DNC IDs) that accounts for the IPU’s network proximity. We discuss that in the next section.
Our microbenchmarks reveal that proximity matters. Specifically, tile
proximity directly affects the latency between pairs of tiles in on-chip communication, and network proximity between IPUs directly affects inter-IPU
latency.
Neighboring IPUs experience the lowest round-trip latency when communicating with each other, whereas marginal latency increases progressively
when the endpoint IPUs are farther and farther away. For example, the two
IPUs on a board can typically perform a minimum data transfer in 0.63...0.83
microseconds, and two neighboring IPUs of same parity can perform a transfer in 0.54...0.77 microseconds; any addition hop costs on average 0.16 microseconds (varying between 0.145 and 0.174); we present these results in Section 4.1.3 and Figure 4.4. IPUs 7 and 10 are the farthest pair and experience alatency of approximately 1.76 microseconds.

20

CHAPTER 2. EXPERIMENTAL SETUP

37

38

113

114

189

190

36

39

112

115

188

191

...

...

...

3

72

2

73

1

74

77

150

0

75

76

151

Column 0265

Column 1

152

266

341

...

227

Column 2

228

342

417

...

303

Column 3

304

418

493

...

379

Column 4

380

418

569
...

...

455

Column 5

456

570

455

Column 6

533

606

532

607

Column 7

IPU Exchange

Column 15

Column 14

Column 13

Column 12

Column 11

Column 10

Column 9

Column 8

1140

1215

1064

988

912

836

760

684

608

683

1141

1214

609

682

1142

1213

1143

1139

1063

987

911

835

759

1212
...

...

1176

1179

1177

1178

1101

...

1102

1025

...

1026

949

...

950

873

...

874

797

...

798

721

...

722

644

647

645

646

Figure 2.2: Detailed tile layout within an IPU processor, including tile numbering. The logicaltile IDs depicted are those exposed to the programmer. Tiles topologically close to each otheractually experience shorter communication latencies than tiles farther away (see Section 4.1.2).
Source: direct communication with Graphcore.

On the other hand, peak bandwidth between two endpoint IPUs is notaffected by their proximity (Figure 4.6) as the topology suggests. Also in accordance with topology are our findings on bidirectional communications between the IPUs on each board: we achieved 108 GB/s, or 84% of the nominal
peak bandwidth (Section 4.1.6, Table 4.6). We find that monodirectional bandwidth achieves almost exactly half that much bandwidth.
The cursory results we just listed are only examples of our findings; thenext chapter is entirely dedicated to studying the IPU’s interconnect performance in diverse communication patterns and under different loads, as measured with our benchmarks.
Tile numbering. The on-chip interconnect among tiles within an IPU isrepresented in Figure 2.2, where the actual logical tile ID numbering is shown.
Tiles are arranged in columns containing 76 total tiles. These 76 tiles are organized in 19 islands containing four tiles each. Our benchmarks shows thatproximity affects latency and that neighboring tiles within the same island
experience a marginally shorter latency between each other than tiles further away within the same column. Tiles in different columns experience a
marginally higher latency.
A developer who is strongly focused on latency could, at the extreme,
manually place compute vertices on tiles in order to take advantage of the2.2. MULTI-IPUS

21

respective proximities.

2.2

Multi-IPUs

This section discusses how the platform groups physical IPUs into virtualMulti-IPU devices and how IPU tiles are numbered within each Multi-IPU
device. All details reflect the concrete configuration of the Graphcore evaluation system we benchmarked.
On our evaluation system, the drivers and SDK expose the exact set ofMulti-IPU devices of Figure 2.3. Each Multi-IPU possesses a unique Device
ID; their numbering resumes from where physical ID numbering ends. On
our system, the highest-numbered physical IPU has Device ID 15, and thefirst Multi-IPU (virtual) has Device ID 16.
For performance reasons, Multi-IPU are constructed only out of neighboring IPUs and only in powers of two (2, 4, 8, and 16 IPUs). All 2-IPU devices
contain a pair of IPUs residing on the same board. For example, IPU16 contains IPU7 and IPU5. All 4-IPU devices contains the IPU from a pair of C2
cards that are neighbors in the ladder network topology, and so on.
Multi-IPUs correspond to partitions of the following sequence of Device
IDs into equally-sized, contiguous sub-sequences with power-of-two lengths:

5, 7, 4, 6, 3, 1, 2, 0, 13, 15, 12, 14, 9, 11, 10, 8.
Trivially, this sequence corresponds to a front-to-back, left-to-right enumeration of IPUs in the chassis. For the complete avoidance of doubt, chassis front
and back are denoted in the picture by the Fwd and Aft arrows, whereas left
and right are as seeon by an observer placed at the back of the chassis andlooking forward (Fwd direction).
Within each Multi-IPU device, the run-time numbers IPUs according to
the sequence just reported. This 0-based index of each physical IPU within a
Multi-IPU is called a DNC ID. For example, IPU16 is composed of IPUs 5 and 7(5 and 7 are their Device IDs); their DNC IDs are trivially 0 and 1, respectively.
Consider IPU30, the device containing all IPUs in the system, whose Device
IDs are 5, 7, 4, 6 ... 10; their respective DNC IDs are 0,1,2,3 ... 15.
To avoid confusion between Device IDs and DNC IDs, the reader only
needs to remember that:• Device IDs reflect PCI device numeration; consecutive Device IDs do
not indicate IPU proximity;

22

CHAPTER 2. EXPERIMENTAL SETUP

C2 board
IPU7

IPU5

IPU16
IPU24

C2 board
IPU6

IPU4

IPU17
IPU28

C2 board
IPU1

IPU3

IPU18
IPU25

C2 board
IPU0

IPU2

IPU19
IPU30

C2 board
IPU15

IPU13

IPU20
IPU26

C2 board
IPU14

IPU12

IPU21
IPU29

C2 board
IPU11

IPU9

IPU22
IPU27

C2 board
IPU8

IPU10

IPU23

Figure 2.3: Topology of Multi-IPU virtual devices (depicted in red) with respect to the physical
IPUs (in black). All numbers are Device IDs. For example, IPU16 is a virtual device containing
physical IPUs 5 and 7. IPU30 is a virtual device containing all 16 physical IPUs in the server.

2.3. METHODS

23• DNC IDs reflect actual IPU proximity in the IPU Link topology; consecutive DNC IDs denote IPUs that are close to each other.
DNC IDs matter to the final user because they also reflect the order inwhich tiles are ordered within the Multi-IPU. Tiles are ordered sequentiallyaccording to the DNC IDs: tiles 0...1,215 belong to IPU with DNC ID 0, tiles
1,216 ... 2,431 belong to DNC ID 1, and so on.
A user employing a different system than the one we evaluated can obtain an explicit enumeration of all IPU devices (physical, and virtual) andtheir topology, including DNC numbering, by using command line tool gcinfo--list-all-devices.

2.3

Methods

Software SDK version. At the time this report is written, the IPU is a novel
architecture and Graphcore is refreshing its Poplar SDK and its IPU drivers
with relative frequency. Because subsequent releases include incremental optimizations, the choice of any one SDK version in conjunction with a benchmark affects the benchmark’s performance results. We used SDK version
1.0.49. Researchers intending to duplicate our results should employ the sameversion.
Whenever meaningful, we timed the latency of each operation with carefor the following concerns:• Single-IPU measurements. In benchmarks involving a single IPU,
we timed operations on the IPU, via the cycle-accurate primitive
popsys::cycleStamp(). The use of this primitive isolates measurementsfrom CPU-IPU communication overheads and IPU program launchoverheads.
• Multi-IPU measurements. Benchmarks involving multiple IPUs can
not practically use popsys::cycleStamp(). We time them instead fromthe host system. We time multiple iterations of each benchmark in orderto amortize overheads, making their impact negligible. To remove spurious overhead between one iteration and the next, we use the Poplar
primitive program::Repeat(), which excludes the host from having any
role in the repetition.
• Warm up. Whenever we desire steady-state measurements, we take appropriate countermeasures to exclude warm-up overheads from the results. For example, whenever benchmarks involve the host, we typically
execute an untimed warm-up iteration of the benchmark before we starttiming.

24

CHAPTER 2. EXPERIMENTAL SETUP

Units. In this paper we prefer harmonized ISO/IEC 80000-13:2008 standard
to denote data sizes and capacities: 1 KiB = 1,024 bytes; 1 MiB = 1,024 KiB; 1
GiB = 1,024 MiB; 1 TiB = 1,024 GiB. When units of capacity are used to express
throughputs, we use customary 1,000-based prefixes (k,M,G,T —thus, kB/s,
MB/s, GB/s, TB/s) for consistency with the literature and ease of calculation.

Chapter 3

Local Memory
We start our analysis from the basic constituents of an IPU system and proceed outward. In this chapter, we focus on the performance of the memories
located within each tile.
On an IPU, each tile possesses 256 KiB of memory that it can access directly
via instructions. (For a tile to access memory that is local to another tile, it
must use the exchange; we characterize the exchange’s performance in thenext chapter.)
We find that the performance of each local memory is fixed (6 cycles latency, 31.1 TB/s aggregate peak read bandwidth); local memories are completely decoupled from each other, in function and performance, as architectural considerations suggest. Pressure on the local memory in one tile does
not affect memory performance in any other tile.

3.1

Latency

The latency experienced by a tile reading a value from its local memory is 6
clock cycles. Our experiments show that this latency is fixed and does notdepend on access patterns, stride, working set size, number of threads usedon each tile (1 ... 6), number of concurrent tiles running the same benchmark
simultaneously, or size of the target working set accessed by the benchmark.
A minimal benchmark suitable to demonstrate latency invariance follows,
courtesy of Graphcore. The code implements a pointer chase that scans anarray of configurable size. The array is pre-populated with indices that realize
a linked-list visit with configurable stride.
template <const int unroll_factor>
class PChaseVertex : public poplar::Vertex {public:poplar::Input<Vector<uint32_t>> in;poplar::Input<uint32_t>
start;
poplar::Input<bool>
flag;

25

26

CHAPTER 3. LOCAL MEMORY

poplar::Output<uint32_t>

out;bool compute() {uint32_t index = start;for (int i = 0; i < unroll_factor; i += 1)
index = in[index];
if (flag)*out = index;
return true;}};
template class PchaseVertex<1000>; // explicit instantiation

In the listing, variable flag presents the compiler with a possible side effect; its purpose is to prevent the compiler from detecting the entire benchmark as dead code and optimizing it out.
Experimental results confirm that neither size nor stride affect memory
access latency.

3.2

Bandwidth

We measure the aggregate read bandwidth available to user code on the entire
IPU and compare it with theoretical limits derived from hardware specifications (31.1 TB/s). We achieved bandwidths closely matching theoretical values only with a benchmark written in assembly language containing a zerooverhead loop of 128-bit loads. Experiments show that Poplar load-dense
code with narrower loads, 32- and 64-bit wide, access roughly a quarter andhalf of the theoretical bandwidth, respectively. Code with lower-than-perfectload instruction density may achieve even lower bandwidth. However, naive
code consisting of array-based, single-precision read loops without hand optimizations emits relatively dense 32-bit loads, achieving a quarter of the theoretical limit. Finally, we show how developers can use the float2 vector types
to increase access width, roughly doubling their bandwidth, without resorting
to assembly code. All these results are presented in detail in this section.
Theoretical limit. The theoretical aggregate read bandwidth derives from thefollowing assumptions: each tile can read 16 bytes per clock cycle, the clock
frequency is 1.6 GHz, and the tile count is 1,216. The result, 31.1 TB/s, is the
product of these three factors. (Top row in Table 3.1.)
Multi-threading. All benchmarks in this section use six identical threads pertile in order to achieve complete hardware thread occupancy. Using a smaller
amount of threads yields proportionally lower bandwidth.
Naive code. To measure the performance limits of loop-based code written
in Poplar/C++, we use a benchmark specifically designed to generate longsequences of load instructions. This benchmark delivers 7.59 TB/s, roughlya quarter of the theoretical limit. (Bottom row in Table 3.1.) Real-world user

3.2. BANDWIDTH
Approach
Theoretical limit

27
Language

Load width(bits)–

Bandwidth(TB/s)

Fraction ofTheoretical

31.13

100 %

Best actual

assembly

128

30.70

98.6%

float264-bit loads
float4

Poplarassembly
Poplar

64
64
64

15.30
15.26
15.02

49.2%
49.0%
48.3%

Naive, float (upper limit)

Poplar

32

7.59

24.4%

Table 3.1: Theoretical and actual aggregate read bandwidths available on the entire IPU chip, as
measured via diverse benchmarks written in assembly or in Poplar.

code will see even lower performance than 7.59 TB/s if it exhibits a lower
load instruction density. An excerpt of the benchmark source code follows,
courtesy of Graphcore.
#define UNROLL 256
class AccaddVertex32 : public poplar::Vertex {public:Input<Vector<float>>in;poplar::Input<uint32_t>
size;bool compute() {for (int i = 0, e = size ; i < e; i += UNROLL) {float* base = &in[i];#pragma unroll UNROLL
for (int j = 0; j < UNROLL; j++)asm volatile("" :: "r"(base[j]));}return true;}};

The listing contains the array-based loop, with an inner loop containing
an inline assembly block accompanied by an unroll pragma directive. The
inner loop instructs the compiler to generate an unrolled sequence of load
instructions.
We verify by inspection that the corresponding emitted code contains anunrolled sequence of 256 load instructions, each 32-bit wide.
Best actual bandwidth. To determine the absolute highest bandwidth achievable by user code, we benchmark hand-written assembly code that featureszero-overhead loops of 128-bit wide memory accesses. The details of this codeare beyond the scope of this document. It achieves a bandwidth closely matching the theoretical: 30.7 TB/s, or 98.6% of the theoretical limit. (“Best actual”
in Table 3.1.) We know of no other means to emit 128-bit load instructions
than resorting to assembly.
Vector types. Applications loading 32-bit words can achieve the higher 64-bit
bandwidth by loading two words at a time if the source data is contiguous.

28

CHAPTER 3. LOCAL MEMORYThis avenue benefits developers who desire higher performance but are notwilling to write assembly code. They can instruct the compiler to generate
wider loads by using vector type float2 and explicitly aligning their input
arrays, as we illustrate in the listing below. Vector type float2 expresses anarray of two 32-bit floating point values intended to be handled together by
the hardware. This code will emit 64-bit load instructions and achieve thesame performance as a hand-written assembly loop of 64-bit loads (rows labeled “float2” and “64-bit loads” in Table 3.1), which is roughly half of thetheoretical limit.
#define UNROLL 256
class AccaddVertex64 : public poplar::Vertex {public:Input<Vector<float, VectorLayout::SPAN, 8>> in;poplar::Input<uint32_t>
size;bool compute() {float2 *f2in = reinterpret_cast<float2 *>(&in[0]);for (int i = 0, e = size / 2; i < e; i += UNROLL) {float2* base = &f2in[i];#pragma unroll UNROLL
for (int j = 0; j<UNROLL; j++)asm volatile("" :: "r"(base[j]));}return true;}};

The crucial aspects of this listing are the alignment directives associated with
input tensor in and the reinterpret_cast<> syntax necessary to access in via
vector type float2. The inline assembly loop at the bottom is not part of thetechnique we propose, and it is just intended to achieve perfect load instruction density (forcing the compiler to generate an unrolled sequence of load
instructions), as in an example we presented earlier.
This vector type approach does not extend to float4, which does not cause
the compiler to emit 128-bit loads. We find that a benchmark based on float4
roughly achieves the same performance as the code just illustrated (“float4”
in Table 3.1). We omit its listing.
Sensitivity to block size. We show how the aggregate local memory bandwidth seen by a workload varies according to the size of blocks it accesses.
Our benchmark scans a block of configurable size. We instantiate the workload six times per tile to optimally occupy the six hardware threads. We vary
block sizes from a single word to 200 KiB, which is close to the user-available
capacity. Our results are in Figure 3.1; for sufficiently large blocks, the bandwidth saturates smoothly. Accessing memory in blocks of 8 KiB is sufficientto achieve 95% of peak bandwidth.
No congestion occurs as more and more tiles use the respective memories.
This finding is consistent with the operation of completely distributed local
memories.

3.2. BANDWIDTH

29

32
30

Aggregate read bandwidth (TB/s)

28
26
24
22
20
18
16
14
12
10
8
6
4
2
0

0

5

10

15

20

Size of the block accessed on each tile (KB)
float poplar
float2 poplar

float4 poplar
128-bit assembly

64-bit assembly
theoreticalFigure 3.1: Aggregate local memory bandwidth on the entire IPU as a function of block size.

Write bandwidth. The theoretical limit is 15.5 TB/s, derived from each tile’s
ability to write 8 bytes per clock cycle. Naive, array-based, single-precision
write loops achieve the same performance as the corresponding read loops(“float poplar”) in Figure 3.1. That code benefits from the use of vector types
in the same manner as illustrated above and with the same performance benefits. The following listing, courtesy of Graphcore, illustrates how to instruct
the compiler to emit 64-bit wide, aligned write instructions for single precision(32-bit) elements.
#define UNROLL 8
using namespace poplar;class AccaddVertex : public poplar::Vertex {public:Input<Vector<float, VectorLayout::SPAN, 8>> in;// Request 8-byte alignment (SPAN) for variable ‘in’
poplar::Input<uint32_t>
poplar::Input<bool>
poplar::Output<float>

size;
flag;
out;

bool compute() {float2 *f2in = reinterpret_cast<float2 *>(&in[0]);float tmp = 0.0;

30

CHAPTER 3. LOCAL MEMORY// loop limit is size/2 because each element consists of 2 floats
for (int i = 0; i < size/2; i+= UNROLL ){float2 tmps[UNROLL];#pragma unroll UNROLL
for (int j = 0; j < UNROLL; j++)tmps[j] = f2in[i+j];
if (flag)for (int j = 0; j < UNROLL; j++)tmp += tmps[j][0] + tmps[j][1];}
if (flag)*out = tmp;
return true;}};

Using an unroll factor equal to 8 achieves the highest bandwidth. This codeachieves the same bandwidth as the corresponding code that we illustrated
above for array reads.

Chapter 4Interconnect
We evaluate the IPU interconnect’s performance by benchmarking point-topoint and collective operations. We analyze whether and how latency andbandwidth degrade as the scale of the communication operation increases,
involving on-chip and off-chip interconnects.
We selected a sufficiently broad set of primitives for benchmarking that
represent communication patterns commonly found in parallel applications.
Our results are intended to help software designers derive early performance
estimates for their applications.
Our choice of microbenchmarks and metrics conforms with publicly available benchmarking suites for parallel computing, such as the OSU Microbenchmarks[8] by the Ohio State University’s Network-based Computing Laboratory.
The first section of this chapter focuses on point-to-point transfers; the remaining sections each focus on one collective operation.

4.1

Point-to-point Transfers

In this section, we study the performance of IPU systems when engaged inpoint-to-point communications (data transfers from one source tile to one destination tile) under diverse load conditions.
Our results draw a map of the relative proximity of the constituents of an
IPU system (specifically, how quickly tiles can reach each other in terms oflatency and bandwidth, depending on relative distance in the system) that ispredictable and consistent with the known topology of the system presented
in Figures 2.1 and 2.2.
Software designers can take advantage of our results to gain an understanding of the latency penalty and bandwidth associated with traversing the31

32

CHAPTER 4. INTERCONNECT

on-chip interconnect, crossing IPU links on a board, and crossing IPU links
across boards.
We benchmark latency and bandwidth in a spectrum of congestion conditions ranging from global silence (the system is idle, with the only exceptionof the one transfer benchmarked) to full load (all tiles are engaged in communication with the same pattern as the one benchmarked).
(a) Transfer between tiles on the same IPU(b) Transfer between tiles on different IPUs
on the same board(c) Transfer between tiles on different IPUs
on different boards, direct IPU link(d) Transfer between tiles on different IPUs
on different boards, no direct IPU link

IPU board 0

IPU board 0

IPU board 0

IPU board 0

IPU0

IPU1

IPU0

Tile 0

IPU1
Tile 0Tile 1

Tile 0Tile 1

...

...

...

Tile j

IPU
Link

...

...

Tile 1,215

Tile j...

Tile 1,215...
Tile j...
Tile 1,215

Tile i...
Tile j...
Tile 1,215...
Tile i...
Tile jTile 1...
Tile i...

...
Tile jTile 0Tile 1...
Tile i

IPU1
Tile 0Tile 1...
Tile i

IPU0
Tile 0Tile 1...
Tile i

IPU1
Tile 0Tile 1...

Tile i

IPU0

Tile j...
Tile 1,215...
Tile 1,215

IPU
Link

Tile 1,215

IPU
Link

IPU board 1

IPU board 1

IPU0

IPU1
Tile 0Tile 1...

IPU0
Tile 0Tile 1...
Tile iTile iTile j...

Tile 1...

Tile iTile iTile j...

Tile 1,215

IPU
Link

...

...
Tile j...
Tile 1,215

Tile 0Tile 1
...

...

...

IPU1
Tile 0Tile j...

Tile 1,215

Tile 1,215

Figure 4.1: Communication topologies we evaluate in point-to-point latency benchmarks. Datatransfers between the source and the destination tiles are depicted with blue arrows. The experiments are designed to exercise both on-chip (a) and off-chip (b,c,d) interconnects.

4.1.1

Congestion-free Latency

We study the latency incurred by a single, minimum-size transfer between
two tiles while the rest of the system is idle (global silence on the interconnects).
The latencies we measure are consistent with the topology of the experiment. On chip, a source tile can reach any destination tile on the same IPU
within 0.13 microseconds on average. As soon as a message needs to cross an
IPU link to reach its destination, a penalty of approximately 0.5 microseconds
applies. Surprisingly, reaching a tile on the second IPU located on the sameboard as the source is marginally more expensive than reaching a directly connected IPU on a different board. Detailed results are in Table 4.1.
Detailed descriptions of the experimental topologies we benchmarked follow, matching the corresponding illustrations in Figure 4.1 and corresponding
results of Table 4.1:

4.1. POINT-TO-POINT TRANSFERS

33• in experiment (a) we exercise the on-chip interconnect; the source anddestination tiles (i and j) reside on the same IPU. We obtain the averagelatency value over a large number of experiments. In each such experiment, i and j are chosen randomly;• in experiment (b) we exercise the off-chip IPU Link interconnect between two IPUs on the same board;• in experiment (c) we exercise the off-chip IPU Link interconnect between
two IPUs on different boards when those IPUs are directly connected via
an IPU link. Surprisingly, this latency is marginally lower than that ofexperiment (b);• in experiment (d) we exercise the off-chip IPU Link interconnect in conditions where the two IPUs involved are not directly connected via an
IPU link and communication requires traversing more than one IPU
link.
ExperimentOn-chip
Off-chip, on board
Off-board, direct IPU link
Off-board, indirect IPU link

Latency
(a)
(b)
(c)
(d)

0.133 µs
0.633 µs
0.524 µs
0.779 µs

Table 4.1: Point-to-point latency for small messages under no load. Experiments labeled (a)...(d)correspond to the illustration in Figure 4.1 and are described above.

4.1.2

Latency between Tiles on a Chip by Proximity

We study how the physical proximity between pairs of tiles on a chip affects
their communication latency in congestion-free conditions.
We measure latency between all pairs of tiles and depict our results in
Figures 4.2 and 4.3. The first figure focuses on tiles within a column; the second shows the entire IPU. In both figures, the main diagonal (in white) corresponds to local transfers, which are carried out in local memory and do not
involve the exchange. Latencies reflect tile topology (Figure 2.2).
Latency measurements. The reader should pay attention to the fact that thelatency measurements presented within this section may differ from those ofother sections, e.g., Table 4.1. The discrepancy is due to the use of different
timing methods. Specifically, experiments in this section use the on-IPU, finegrained profiling facilities offered with the Poplar SDK (that we described under “Single-IPU measurements” in Section 2.3) which do not include the costof the on-IPU synchronization phase that precedes the transfer, whereas othersections typically use host-based “Multi-IPU measurements” that include thatcost. Both measurements are meaningful when appropriately characterized.

34

CHAPTER 4. INTERCONNECT
75

65

60

55
Latency (ns)Destination tile index

57

38

50

45
19

40

35
0
0

19

38

57

75

Source tile index

Figure 4.2: Minimum latency between all tiles belonging to the same Column on an IPU processor(we first presented columns in Figure 2.2).

Within a column. Data transfers within the same column take 37...59 ns(59...95 clock cycles); see Figure 4.2. Intra-column latency primarily depends
on what island the destination tile belongs to; transfers to the same island havesame latency, no matter the source island. Latency is minimal when transferring to a tile belonging to the island closest to the IPU Exchange (e.g., forColumn 0, that’s island of tiles 0, 1, 74 and 75). Latency increases by 1.25 ns (2clock cycles) for every island the destination moves away from the exchange.
Across columns. Latency is 98 ns (the highest) in transfers from Column 8 to
0 (tile 646 to 0). Transfers from the rightmost columns (column 7 and 8) to theleftmost columns (0 and 15) take more than 63 ns. Transfers in the opposite
direction have lower latency.

4.1.3

Latency between IPUs by Proximity

We measure and chart the minimum latency between pairs of IPUs acrossthe entire 16-IPU experimental system. Latencies correlate directly with the
distance between source and destination IPUs along the network topology of
Figure 2.1. We chart our results in Figure 4.4.
Numbering. In this section we only refer to IPUs by DNC IDs (as opposed to4.1. POINT-TO-POINT TRANSFERS

35

1140
1064

80

988
912

70

760
684

Latency (ns)Destination tile index

836

60

608
532
456

50

380
304
228

40

152
76
0
0

76 152 228 304 380 456 532 608 684 760 836 912 988 1064 1140

30

Source tile index

Figure 4.3: Minimum latency between all pairs of tiles on an IPU processor.

Device IDs—recall the distinction from Section 2.2). This is precisely because
DNC IDs account for the proximity between pairs of IPUs, whereas Device
IDs don’t.
Results. We find that:• the lowest latency is between IPU with DNC IDs i and i ± 2, i.e., eacheven-numbered IPU reaches the fastest those even-numbered IPUs thatare facing it on the neighboring boards. The same applies to oddnumbered IPUs. These links are visible in the matrix as the two diagonallines with the most intense shade of green; they are two cells above themain diagonal and two cells below the main diagonal;• pairs of IPUs located on the same board see the second-best latencies
across the system;• there is no wrap-around. The IPUs with DNC IDs 0 and 14 are not neighbors; neither are 1 and 15. These four IPUs suffer from edge effects inthe sense that they only have one neighbor of same parity. This reflects
the IPU Link network shape (i.e., not a torus).

36

CHAPTER 4. INTERCONNECT

1.8
15 1.760 1.557 1.577 1.414 1.433 1.293 1.310 1.144 1.167 1.028 1.048 0.880 0.904 0.764 0.823
14 1.499 1.733 1.383 1.546 1.238 1.430 1.117 1.281 0.973 1.167 0.854 1.020 0.708 0.902
13 1.588 1.383 1.406 1.237 1.259 1.122 1.141 0.972 0.995 0.856 0.874 0.710 0.800
12 1.357 1.584 1.235 1.401 1.090 1.284 0.970 1.136 0.826 1.020 0.706 0.873
11 1.462 1.238 1.255 1.089 1.112 0.970 0.993 0.824 0.848 0.708 0.767

0.826
0.969 0.766

1.6

0.798 0.706 0.942
0.911 0.707 1.084 0.880
1.4

10 1.193 1.414 1.061 1.229 0.916 1.109 0.797 0.963 0.652 0.846

8 1.032 1.266 0.915 1.083 0.777 0.961 0.650 0.815
7 1.119 0.914 0.932 0.768 0.790 0.650 0.710
6 0.861 1.091 0.742 0.908 0.595 0.790

0.911 0.708 1.062 0.853 1.230 1.029
0.742 0.651 0.885 0.829 1.057 0.972 1.209

1.2

0.856 0.651 1.030 0.824 1.175 0.989 1.350 1.142
0.711 0.652 0.883 0.796 1.031 0.972 1.206 1.116 1.351
1

5 0.946 0.742 0.761 0.595 0.686
4 0.713 0.945 0.596 0.761
3 0.800 0.595 0.656
2 0.540 0.772
1 0.6310.853 0.652 1.003 0.798 1.174 0.972 1.325 1.118 1.495 1.294
0.685 0.595 0.828 0.773 1.004 0.915 1.147 1.090 1.323 1.239 1.469

0.800 0.594 0.971 0.767 1.121 0.916 1.293 1.089 1.440 1.234 1.614 1.411

0.8

0.656 0.594 0.827 0.740 0.975 0.915 1.148 1.064 1.296 1.238 1.467 1.382 1.615
0.800 0.597 0.946 0.743 1.119 0.917 1.269 1.063 1.440 1.236 1.589 1.384 1.760 1.556
0.6

00.630 0.539 0.773 0.712 0.945 0.859 1.092 1.035 1.266 1.180 1.415 1.351 1.587 1.499 1.734
0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Source IPU index

Figure 4.4: Minimum latency between each pair of IPUs in our experimental 16-IPU system,measured in zero-congestion conditions. IPUs are numbered according to their DNC IDs, as
discussed in Section 2.2.

Latency (us)Destination IPU index

9 1.269 1.062 1.082 0.916 0.938 0.799 0.818 0.652 0.743

0.770 0.708 0.938 0.852 1.087

4.1. POINT-TO-POINT TRANSFERS

4.1.4

37

Latency under LoadIn this section, we investigate the impact of congestion on point-to-point latency. We simulate traffic conditions by instantiating identical point-to-pointtransfers that occur concurrently.
We compare latencies in the congestion-free scenario and under load in
Table 4.2. In the table, experimental conditions (a)...(d) refer to same topologies we considered in our congestion-free study (Section 4.1.1). In summary,congestion increases on-chip latency only marginally (+24%), but it affects offchip IPU link latency significantly, with slowdowns of 4.0...7.7×.
We also find that the average per-message latency in a multi-IPU system isremarkably scalable: randomized concurrent transfers across a 16-IPU system
experience a latency that is minimally higher (1.93 ns/message) than on asingle-board 2-IPU system (1.76 ns/message).
The methods we used to put the interconnect under load are described
below.
ExperimentOn-chip
Off-chip, on board
Off-board, direct IPU link
Off-board, indirect IPU link

(a)
(b)
(c)
(d)

Congestion-free
Latency

Latency
under load

Congestion
degradation

0.133 µs
0.633 µs
0.524 µs
0.779 µs

0.165 µs
2.521 µs
2.524 µs
5.989 µs

1.2×
4.0×
4.8×
7.7×

Table 4.2: Effect of congestion on point-to-point latency. This is a short summary that compares
congestion and no-congestion results in conditions (a)...(d) corresponding to the previous section.
More detailed experiments are in the next table.

Experiments (a)...(d) in presence of congestion match the corresponding
congestion-free experiments (described in the previous section), with the following additional details:• in experiment (a), each tile on an IPU performs one transfer to one randomly selected tile on the same IPU; as many transfers occur concurrently as there are tiles on one IPU (1,216). Each tile participates in exactly one transfer as a source and in exactly one (other) transfer as adestination;• in experiment (b), we exercise the off-chip IPU Link interconnect between two IPUs on the same board. Each tile on one IPU performs
one transfer to one randomly selected tile on the other IPU on the sameboard. As many transfers occur concurrently as there are tiles on one
IPU. Each tile on the first IPU participates in exactly one transfer as asource, and each tile on the second IPU participates as a destination. All
cross-IPU traffic is monodirectional;

38

CHAPTER 4. INTERCONNECT• in experiment (c), we exercise the off-chip IPU Link interconnect between two IPUs on different boards when those IPUs are directly connected via an IPU link (e.g., each tile on IPU5 performs one transfer to
one randomly selected tile on IPU4; see Figure 2.1). As many transfers
occur concurrently as there are tiles on one IPU. Each tile on the source
IPU participates in exactly one transfer as a source, and each tile on thedestination IPU participates in exactly one transfer as a destination. All
cross-IPU traffic is monodirectional;• in experiment (d) we exercise the off-chip IPU Link interconnect in conditions where the two IPUs involved are not directly connected via an
IPU link and communication requires traversing more than one IPU link(e.g., each tile on IPU5 performs one transfer to one randomly selected
tile on IPU6; see Figure 2.1). All other conditions are the same as inexperiment (c).

For a finer-grained characterization of congestion impact, see the augmented results of Table 4.3 and our observations that follow.
Scale(IPUs)

1/8
1/4
1/2
1
2

2
2
2
4
8
16

Experiment

Concurrent
transfers

Total
latency

Avg. latency
per message

On-chip: source and destination tiles are on the same IPU
1
2
4
8
...
16
...
38
...
76
...
152
... a quarter of one IPU
304... half IPU
608... the entire IPU
(a)
1,216

0.123 µs0.130 µs
0.157 µs0.152 µs0.160 µs
0.148 µs
0.165 µs0.163 µs
0.165 µs
0.165 µs
0.165 µs122.565 ns
65.040 ns
39.253 ns
18.994 ns
10.009 ns
3.884 ns2.173 ns
1.070 ns
0.541 ns0.272 ns
0.136 ns

Cross-IPU experiments - 2 IPUs - monodirectional traffic... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link
(d)
1,216

2.521 µs
2.524 µs
5.980 µs

2.073 ns
2.076 ns
4.917 ns

Cross-IPU experiments - random system-wide destinations... both IPUs on the same board
2,432
4.282 µs... across boards, direct IPU link
2,432
2.572 µs... across boards, no direct IPU link
2,432
6.030 µs
4 IPUs on two boards
4,864
5.981 µs
8 IPUs on four boards
9,728
16.532 µs
16 IPUs, entire system
19,45637.611 µs

1.761 ns
1.058 ns
2.480 ns
1.230 ns
1.699 ns
1.933 ns

Table 4.3: Fine-grained and larger-scale detailed results on the effect of congestion on latency.

In the on-chip experiments, we vary congestion by varying the number ofconcurrent transfers from 1 to 1,216. In each transfer, a randomly selected setof tiles (of count 2, 4, 8 ... 1,216) transfer to as many destination tiles on the4.1. POINT-TO-POINT TRANSFERS

39

same IPU. At the upper extreme, this coincides with experiment (a) as alreadydescribed.
The middle band of the table contains results of experiments (b), (c) and (d)involving 2 IPUs where the source tiles are segregated on one IPU and all thedestination tiles are on another. All cross-IPU traffic in these experiments ismonodirectional. We have described these experiments earlier in this section.
In the bottom band of the table, we present results corresponding to desegregated destinations. We consider varying system scales (from 2 to 16 IPUs)
and, at every scale, all the tiles in the system are the source of one transfer.
The destination of each transfer is chosen randomly among all tiles in the system. The destination tile can be on the same IPU as the source or on any other
IPU. This communication scheme is representative of parallel applications using a domain decomposition that requires uniformly spread communication.
Results show good scalability across system sizes. The average per-message
latency does not seem to grow significantly in a 16-IPU system compared tosmaller systems.

4.1.5

Congestion-free Peak Bandwidth

We study the peak bandwidth available to a single point-to-point transfer between pairs of tiles in different topologies in congestion-free conditions (noother operations occurring in the rest of the system; global silence on all interconnects except for the transfer being benchmarked).
Peak. Everywhere in this paper, the term peak bandwidth denotes the bandwidth seen by transfers of sufficiently large messages (we use the terms blockand message equivalently). On most interconnect architectures, peak bandwidth values are achieved with larger messages. That happens because transfers both of large and small blocks tend to incur similar communication setup
overheads, but with larger blocks those overheads are amortized on higher
byte counts, thus yielding higher throughput values. This property holds on
the IPU too, for both the on-chip IPU interconnect and the off-chip IPU Link
network. We verify this claim experimentally in Section 4.2.3.
Our experiments cover the same topologies already described in Section 4.1.1 and depicted in Figure 4.1.
All results follow in Table 4.4. There is a 14% decrease in point-to-point
bandwidth when moving from on-chip to off-chip, on-board communication.
There is an additional 9% penalty when moving from on-board to inter-board
communication with source and destination IPU linked directly. When IPUs
are not connected directly, bandwidth decrease marginally (less than 2%).
Tile proximity on chip. Physical proximity between the source tile and destination tile on a chip does not affect the peak bandwidth available between
the two tiles. We compare the bandwidth accessible between physically near

40

CHAPTER 4. INTERCONNECT
Experiment

Peak Bandwidth

On chip
Off chip, on board
Off board, direct IPU link
Off board, indirect IPU link

(a)
(b)
(c)
(d)

6.34 GB/s
5.46 GB/s
4.99 GB/s
4.91 GB/s

Table 4.4: Point-to-point peak bandwidth under no load. Experiments labeled (a)...(d) correspondto the topologies in Figure 4.1.

32
Total latency (us)Bandwidth (GB/s)

7

4.2

2.6
1.8
1.4

2

4

8

16

32

Array size (KB)

8
4
2
1

neighbor
distant
1

16

64 1280.5neighbor
distant
1

2

4

8

16

32

64 128Array size (KB)

Figure 4.5: The effect of physical proximity on tile-to-tile transfer bandwidth within an IPU is
negligible, especially with large messages. In our experiments, the chosen pair of neighboringtiles is (0,1) and the chosen pair of far tiles is (0,644), consistent with the tile enumeration of
Figure 2.2.

and far tiles at various block sizes; results are in Figure 4.5. Performance is
indistinguishable for sufficiently large data blocks. When blocks are smaller
than 32 KiB, far tile pairs suffer a slightly longer transfer setup time.

4.1. POINT-TO-POINT TRANSFERS
Experiment

On chip
Off chip, on board
Off board, direct IPU link
Off board, indirect IPU link

41

Congestion-free
Bandwidth(GB/s)

Bandwidthunder load
(GB/s)

Degradation

6.34
5.46
4.99
4.91

6.21
0.04360.02240.02241.02 ×125.2 ×222.8 ×219.2 ×

(a)
(b)
(c)
(d)

Table 4.5: Effect of congestion on point-to-point bandwidth. This is a short summary that compares congestion and no-congestion results in conditions (a)...(d) corresponding to the previous
section. Extended results are available in the next table.

4.1.6

Peak Bandwidth under LoadWe study the peak bandwidth available to concurrent point-to-point transfers between pairs of tiles in different topologies and under different loads.
We consider the same topologies as in the previous sections (Figure 4.1). We
compare our results against congestion-free ones (Table 4.5).
On chip. The aggregate bandwidth available on the on-chip interconnect
scales virtually perfectly with the number of concurrent transfers. As concurrent transfers grow in number, the bandwidth seen by each transfer remains
virtually constant at 6.3 GB/s; see the first band of Table 4.6, culminating inexperiment (a).
Off chip. The off-chip bandwidth offered by IPU links is lower than thatoffered on chip by the exchange. An intra-board IPU link connection offers 55
GB/s in each direction. IPU Link connections between different boards offer
approximately half that much bandwidth per direction, 28 GB/s, regardlessof whether the pair of IPUs is directly or indirectly connected via IPU links.
Bidirectional bandwidth under load effectively doubles the monodirectional
values.
Our extended results of Table 4.6 provide a finer-grained characterizationof congestion.
Experiments (b),(c) and (d) focus on the measurement of monodirectional
inter-IPU bandwidth in the same topologies depicted in Figure 4.1. They all
involve 1,216 concurrent transfers, arranged so that each tile on the source IPU
is the source of exactly one transfer directed at exactly one randomly selected
tile located on the destination IPU.
Results show that the on-board, inter-IPU monodirectional bandwidth isroughly twice as high as across boards (55 vs 28 GB/s).
Additionally, we measure bidirectional inter-IPU bandwidth with experiments (b*),(c*) and (d*), which are the bidirectional extension of (b), (c), and(d), respectively. Experiments (b*),(c*) and (d*) all involve 2,432 transfers, of42

CHAPTER 4. INTERCONNECT

Scale(IPUs)

1/8
1/4
1/2
1
2

2

2
2
2
2
4
8
16

Experiment

Concurrent
transfers

Aggregate
bandwidth(GB/s)

On-chip: source and destination tiles are on the same IPU
One tile to one tile
1
6.34
2 tile to 2 tiles
2
12.65
...
4
25.29
...
8
50.58
...
16
101.06...
38
240.12...
76480.32...
152
959.89... a quarter of one IPU
304
1,919.68... half IPU
608
3,839.22... the entire IPU
(a)
1,216
7,679.01

Bandwidth
per transfer(MB/s)
6,341.
6,323.
6,323.
6,323.
6,316.
6,319.
6,320.
6,315.
6,315.
6,315.
6,315.

Cross-IPU experiments - 2 IPUs - monodirectional... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link
(d)
1,216

55.00
27.72
27.71

45.23
22.79
22.79

Cross-IPU experiments - 2 IPUs - bidirectional... both IPUs on the same board
(b*)
2,432
... across boards, direct IPU link
(c*)
2,432
... across boards, no direct IPU link
(d*)
2,432

108.09
54.86
55.02

44.44
22.56
22.62

Cross-IPU experiments - randomized system-wide destinations - no segregation
to 2 IPUs... both IPUs on the same board
2,432
109.5745.05... across boards, direct IPU link
2,432
54.86
22.56
... across boards, no direct IPU link
2,432
55.02
22.62
to 4 IPUs on two boards
4,864
109.7222.56
to 8 IPUs on four boards
9,728
111.1311.42
to 16 IPUs, entire system
19,456111.28
5.72

Table 4.6: Point-to-point peak bandwidth: bandwidth available to concurrent transfers under load.

which 1,216 go from one IPU to another and 1,216 go in the opposite direction. Each tile on the first IPU is involved in exactly two transfers: in one asa source tile, and in the other as a destination tile. In both such transfers, theother endpoint is on the other tile. The same is true for the second IPU. Alltransfers are across IPUs. No transfers are IPU-local.
These experiments’ results show that bidirectional aggregate bandwidth istwice as high as the corresponding monodirectional values (108 GB/s vs. 55,
and 55 GB/s vs 28).
The bottom band in the table studies per-transfer bandwidth degradation
as the system size scales up. We instantiate Multi-IPUs comprising growing
physical IPU counts (2, 4, 8, and 16), and in each experiment we originate
exactly one transfer from each tile in the Multi-IPU system toward a randomly
selected tile in the system (uniform distribution). Source and destination tiles
may or may not be on the same IPU or C2 board.

4.1. POINT-TO-POINT TRANSFERS

43
60

15

27

27

27

27

27

27

27

27

27

27

27

27

27

27

14

27

27

27

27

27

27

27

27

27

27

27

27

27

27

13

27

27

27

27

27

27

27

27

27

27

27

27

54

12

27

27

27

27

27

27

27

27

27

27

27

27

11

27

27

27

27

27

27

27

27

27

27

54

10

27

27

27

27

27

27

27

27

27

27

9

27

27

27

27

27

27

27

27

54

8

27

27

27

27

27

27

27

27

7

27

27

27

27

27

27

54

6

27

27

27

27

27

27

5

27

27

27

27

54

4

27

27

27

27

3

27

27

54

2

27

27

1

54

54
54
55
27

27

54

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

27

54

27

27

27

27

27

27

27

27

27

27

27

27

27

27

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

45
Bandwidth (GB/s)

Destination IPU index

50

40

35

30

25

0
0

20

Source IPU index

Figure 4.6: Peak monodirectional bandwidth between pairs of IPUs, in congestion-free conditions. IPUs are numbered according to their DNC IDs, as discussed in Section 2.2.

In a 16-IPU system, the average per-tile bandwidth degrades to approximately 5.72 MB/s, which is 25% of the monodirectional aggregate per-tile
bandwidth available when the system involves a pair of IPUs (not on the sameboard).

4.1.7

Peak Bandwidth between IPUs by Proximity

We study how IPU proximity in a multi-IPU system affects monodirectional
peak bandwidth in congestion-free conditions. We find that it does not—a
randomly selected pair of IPUs not located on the same board can transfer
data at 27 GB/s (monodirectional) regardless of their respective position in
the ladder network (Figure 4.6).
Benchmark. Our benchmark considers all (source,destination) pairs of IPUs.

44

CHAPTER 4. INTERCONNECT

We number IPUs according to their DNC IDs. In each experiment, we create
exactly 1,216 transfers. Each tile in the source IPU transfers a 4 KiB message
to a distinct, randomly selected tile on the destination IPU. No other transfers
are occurring in the system.

4.1.8

Multi-threaded Peak Bandwidth

We study the effect of multi-threading on the peak bandwidth available onchip and off-chip. We find the following:• a single thread is sufficient to achieve full bandwidth;• the use of multiple threads is not necessary to achieve peak values;• if used (potentially for other purposes), multi-threading will not cause
any bandwidth degradation.
This investigation is motivated by the fact that tiles support executing instructions from up to six concurrent thread contexts in an SMT-like fashion.
It is legitimate for a software designer to wonder whether the use of multiple
threads to initiate concurrent transfers will achieve a higher aggregate bandwidth.
Results show that the use of additional threads causes no material change
in performance, neither on chip nor across chips.

Experiment

1 thread2 threads

Aggregate bandwidth (GB/s)
3 threads
4 threads
5 threads

On-chip: source and destination tiles are on the same IPU
One tile to one tile
6.14
6.13
6.13
2 tiles to 2 tiles
12.22
12.27
12.24
4 tiles to 4 tiles
24.45
24.52
24.48
8 tiles to 8 tiles
48.96
48.95
48.92
16 tiles to 16 tiles
97.85
97.86
97.87
38 tiles to 38 tiles
232.54232.44232.3776 tiles to 76 tiles
464.62464.59464.32152 tiles to 152 tiles
928.64927.85928.49304 tiles to 304 tiles
1,857.74
1,855.70
1,856.45
608 tiles to 608 tiles
3,713.81
3,712.90
3,712.75
1,216 tiles to 1,216 tiles
7,425.80
7,427.92
7,425.80
Cross-IPU experiments - 2 IPUs - monodirectional... both IPUs on the same board
54.99
54.99
... across boards, direct IPU link
27.71
27.71
... across boards, no direct IPU link 27.71
27.71

54.99
27.71
27.71

6 threads

6.13
12.26
24.54
48.94
97.94
231.96464.76928.491,856.45
3,712.45
7,425.80

6.13
12.07
24.51
48.92
97.89
232.03464.60928.41
1,857.36
3,712.90
7,426.41

6.13
12.29
24.49
48.93
97.76
232.15464.49928.191,856.30
3,702.61
7,428.22

54.90
27.67
27.66

54.88
27.66
27.63

54.91
27.67
27.66

Table 4.7: Multi-threaded point-to-point peak bandwidth: the use of different threads per tile does
not affect the bandwidth available to concurrent transfers.

Experiment scale: 1 IPU

Experiment scale: 2 IPUs

Experiment scale: 4 IPUs(a) Broadcast on chip
IPU board 0
IPU0

IPU1

Tile 0On chip

Tile 1
Tile 2...
Tile 1,215(b) Broadcast to an entire (other) IPU on the same board
IPU board 0

IPU board 0

IPU0

IPU1

IPU
Link

Tile 0Multi-chip,on board(e) Broadcast to both IPUs on a board (including source IPU)

IPU0

Tile 0Tile 1

Tile 1

Tile 1

Tile 2

Tile 2

Tile 2

...

...

...

...

Tile 1,215

Tile 1,215

Tile 1,215

Tile 1,215(c) Broadcast to an entire IPU on a different board,
direct IPU connection
IPU board 0

Tile 0Tile 1
Tile 2(f) Broadcast to 2 entire IPUs, the second IPU being
on a different board; direct IPU connection
IPU board 0

IPU0

IPU1

IPU1

IPU0

IPU
Link

Tile 0Tile 1(h) Broadcast to 4 entire IPUs on two boards
IPU board 0

IPU0

Tile 0

IPU1

IPU
Link

Tile 0Tile 0Tile 1

Tile 1

Tile 2

Tile 2

Tile 2

...

...

...

...

Tile 1,215

Tile 1,215

Tile 1,215

Tile 1,215

IPU
Link

IPU
Link

IPU board 1

IPU1

IPU1

Tile 1
Tile 2IPU
Link

IPU board 1

IPU0

IPU
Link

Tile 0IPU
Link

IPU board 1

IPU0

Acrossboards

IPU1

IPU
Link

Tile 0IPU0Tile 0

IPU1

Tile 0IPU
Link

Tile 0Tile 1

Tile 1

Tile 2

Tile 2

Tile 2

...

...

...

...

Tile 1,215

Tile 1,215

Tile 1,215

Tile 1,215(d) Broadcast to an entire IPU on a different board,
no direct IPU connection
IPU board 0

Tile 1IPU
Link

Tile 1
Tile 2(g) Broadcast to 2 entire IPUs, the second IPU being
on a different board; no direct IPU connection
IPU board 0

IPU0

IPU1

IPU0

Tile 0

IPU1

IPU
Link

Tile 0Tile 1

Tile 1IPU
Link

Tile 2

Tile 2...

...

Tile 1,215

Tile 1,215

IPU
Link

IPU board 1
IPU0

IPU1

Tile 0IPU
Link

IPU board 1

IPU
Link

IPU0

IPU1

IPU
Link

Tile 0Tile 0Tile 1

Tile 1

Tile 1

Tile 0Tile 2

Tile 2

Tile 2

...

...

...

...

Tile 1,215

Tile 1,215

Tile 1,215

Tile 1,215

Tile 1
Tile 2

Figure 4.7: Topologies we benchmark in our broadcast and scatter experiments. Different topologies have different connectivity between source and destination tiles and perform differently. We
depict data flows from the source to the destination tiles with blue arrows. Reversing all blue
arrows depicts the data flows in the gather experiments.

46

4.2

CHAPTER 4. INTERCONNECT

BroadcastThis is the first section in this chapter dedicated to studying the performanceof collective operations. Specifically, this section focuses on latency and bandwidth available to broadcast operations at different scales and for different
topologies.
Broadcast. In a broadcast operation, one tile sends the same message to multiple destination tiles. Each destination tile receives an identical copy of thedata. The source tile maintains a source buffer available to the operation, in
local memory, for the entire duration of the operation. Correspondingly, each
destination tile maintains a local destination buffer available to the operation.
In benchmarks involving whole IPUs, the source tile is also among the destinations, and maintains both a source and a destination buffer in its local
memory. This limits the largest block size available for this benchmark to approximately 100 KiB.
Scale. By scale we mean the destination tile count, or the IPU count if whole
IPUs are involved (Table 4.7). We vary scale from one tile to the entire test
system, which includes 16 IPUs on 8 boards. When using 2 IPUs, we study
the impact of topology and source-destinations proximity: we place the sourceand destinations on the same board (experiment (b)) and on different boards.
We consider pairs of boards directly or indirectly connected with IPU Links(experiments (c) and (d), respectively).
No load. Experiments in this section study load conditions in which no otheroperation is in progress. Each transfer’s performance is affected only by theload caused by the remainder of the collective operation.
Topologies. We benchmark the topologies of Figure 4.7, where thin blue arrows depict the data flows from the source to the destination tiles. Experiments at different scales are illustrated in different columns of the figure: subfigures in the first column illustrate experiments where the destination tiles
belong to 1 IPU; the second column illustrates experiments where destination
tiles span 2 IPUs; and the third column illustrates experiments where the destination tiles span 4 IPUs. We omit experiments involving 8 and 16 IPUs forbrevity.
In the figure, experiments on different rows involve different portions of
the interconnect. In the first row, only the on-chip interconnect is involved.
Experiments in the second row also involve the IPU Link connection between
two IPUs located on the same board. Experiments in the lower rows involve
IPU Links across multiple boards.
Experiments (b) and (e) study the cost of performing a broadcast between
IPUs when both IPUs reside in the same tile. The difference between (b) and(e) is that in (e), the tiles on the first IPU are also broadcast destinations, whilein (b), no tiles of the first IPU are destinations.

4.2. BROADCAST

47

The same difference appears between experiments (c) and (f). Again, thesame difference appears between experiments (d) and (g). This is symbolized
in the picture by additional blue arrowheads in (f) and (g) that point to tiles
on the source IPU.
Experiments (c) and (f) involve 2 IPUs that are not on the same board but
are connected directly via IPU Links. In contrast, experiments (d) and (g)involve 2 IPUs that are not connected directly via IPU Links.

4.2.1

Congestion-free Broadcast Latency

We focus first on the minimum latency associated with a broadcast operation.
To do that, we transfer a message of minimum size, i.e., a word of 32 bits. Wevary the experiment’s scale from one tile to the whole system. We find thatthe system displays remarkable scalability, with a 16-IPU broadcast taking
less than 2 microseconds. Our results are in Table 4.8.
The first experiment in the table (not labeled) is a mere transfer from a tile
to itself. It results in a local memory copy that doesn’t involve the exchangeor IPU links. We report it (12 nanoseconds) only for reference; the reader can
subtract this value from subsequent result values to separate the local andinterconnect contributions to latency.
In the experiments described in the following rows, the set of destination
tiles grows till it reaches the entire IPU.
One tile can broadcast one word to the entirety of its IPU in less than
0.2 microseconds. The additional penalty to perform an off-chip broadcast
involving 2 IPUs is approximately 0.5 microseconds. The total latency of abroadcast targeting all 16 IPUs in the test system is below 2 microseconds.
Observations:• the latency of a broadcast operation involving the entire chip roughly
grows logarithmically with tile count (Figure 4.8, left);• similarly, the latency of a broadcast operation spanning across multiple
IPUs roughly grows logarithmically with IPU count (Figure 4.8, right);• performing a broadcast across 2 IPUs takes approximately 0.7 microseconds; that is approximately 0.5 microseconds more expensive than asingle-IPU broadcast;• the cost of performing an off-chip broadcast is dominated by the costof traversing an IPU Link; the latency we measure in experiments (b)and (e) is identical. This suggests that in experiment (e), the on-chipbroadcast involving the source IPU occurs in parallel with the sequenceof operations comprising the transfer to across the IPU link followedby the on-chip broadcast on the destination IPU. The latter sequence

48

CHAPTER 4. INTERCONNECT

0.2

2
On-chip

Off-chip

0.175

1.75

0.15

1.5

1.25

Latency (us)Latency (us)

0.125

0.1

0.075

1

0.75

0.05

0.5

0.025

1
2
4
8
16

0

38
76
152
304
608
1216

0.25

Destination tile count

0

1

2

4

8

16

Destination IPU countFigure 4.8: Broadcast latency scaling. Left: scaling within an IPU (on chip). Right: scaling andacross multiple IPUs (off chip).

4.2. BROADCAST
Scale(IPUs)

Experiment49
Destination
tile countTotal
latency

10.012 µs

Transfer to self (via local memory)

1/8
1/4
1/2
1
1

2

4
8
16

On-chip: source and destination tiles are on the same IPU
One tile to n tiles
1
0.094 µs...
20.094 µs...
40.094 µs...
80.094 µs...
160.094 µs...
380.134 µs...
760.136 µs...
1520.142 µs... a quarter of one IPU
304
0.153 µs... half IPU
608
0.176 µs... the entire IPU
(a)
1,216
0.194 µs
to an entire IPU, different than that of source tile... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link (d)1,216

0.747 µs
0.637 µs
0.896 µs

to 2 IPUs... both IPUs on the same board... across boards, direct IPU link... across boards, no direct IPU link

(e)
(f)
(g)

2,432
2,432
2,432

0.747 µs
0.637 µs
0.896 µs

to 4 IPUs on two boards
to 8 IPUs on four boards
to 16 IPUs, entire system

(h)
(i)
(j)

4,864
9,728
19,4560.900 µs1.231 µs
1.921 µs

Table 4.8: Broadcast minimum latency: latency necessary to broadcast one value from one tile toa set of tiles. We study destination sets of varying size and location. System scale and respective
location of source and destination tiles affect latency. Experiments labeled (a)-(j) benchmark the
topologies of Figure 4.7.

takes longer than the local broadcast and determines the overall latency.
Intuitively, once the penalty to reach remote tiles across the IPU Link ispaid, the marginal cost of also reaching local tiles is null.

• same considerations apply between pair of experiments (c,f), and pair(d,g);• unexpectedly, a broadcast between two IPUs on the same board (e) takes
marginally more time than between the IPUs located on different boards(f,g). This result is consistent with our gather results (see Section 4.3.1).

50

CHAPTER 4. INTERCONNECT

4.2.2

Peak Broadcast Bandwidth

In this section we study the peak bandwidth available to broadcast operationsof different scales. As in the previous section, we study the effect of sourcedestination IPU proximity in the topologies (a)...(j) of Figure 4.7. Our results
are in Table 4.9.
The message size we employ in these experiments is 100 KiB. It is sufficiently large to saturate the aggregate bandwidth per tile, and at the sametime, it is close enough to the maximum size that allows for two copies (sourceand destination buffer) to exist simultaneously in each tile’s local memory (248
KiB are available to the user). Fitting both buffers in local tile memory allows
us to run experiments where the source tile is also among the destinations.
No other traffic is occurring in the system except for that created by thebroadcast operation itself.
Scale(IPUs)

1/8
1/4
1/2
1
1

2

4
8
16

Experiment

Concurrent
transfers

Aggregate
bandwidth(GB/s)

Bandwidth
per transfer(GB/s)

6.35
12.70
50.49
100.92201.58477.10954.17
1,906.51
3,808.50
7,595.88
15,134.80

6.35
6.35
12.62
12.61
12.60
12.56
12.55
12.54
12.53
12.49
12.46

to an entire IPU, different than that of source tile... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link
(d)
1,216

6,871
6,433
6,427

5.65
5.29
5.29

to 2 IPUs... both IPUs on the same board... across boards, direct IPU link... across boards, no direct IPU link

(e)
(f)
(g)

2,432
2,432
2,432

9,440
9,051
9,034

3.88
3.72
3.71

to 4 IPUs on two boards
to 8 IPUs on four boards
to 16 IPUs, entire system

(h)
(i)
(j)

4,864
9,728
19,45625,43736,75647,343

5.23
3.78
2.43

On-chip: source and destination tiles are on the same IPU
Point to point1
One tile ... to 2 tiles
2... to 4 tiles
4
...
8
...
16
...
38
...
76
...
152
... to a quarter of one IPU
304... to half IPU
608... to the entire IPU (same as source tile)
(a)
1,216

Table 4.9: Peak bandwidth available to broadcast operations of varying scale. Experiments labeled (a)-(j) correspond to the topologies illustrated in Figure 4.7; we describe them in detail.

On-chip performance. As the scale of the broadcast increases within an IPU,
the bandwidth per tile achieved by the operation saturates around 12 GB/s.
12 GB/s seems to be the on-chip average per-transfer peak. Saturation occurs
with small destination sets: a destination set of 4 tiles is sufficient to achieve

4.2. BROADCAST

51

peak per-tile bandwidth.
Off-chip performance. A direct comparison between experiments (a) and (b),that have identical destination count, reveals the penalty for extending thebroadcast off chip: a drop in average per-transfer bandwidth from 12.4 to 5.6
GB/s. The performance difference between on-board and inter-board communication is negligible.
Scaling. The results of experiments (h)-(j) show that larger system scales benefit from a monotonically increasing aggregate broadcast bandwidth. The
growth trend is, however, not linear, and not trivially explained.

4.2.3

Effect of Message Size on Broadcast Bandwidth

We study broadcast bandwidth performance below peak, i.e., when smaller
messages are transferred. We find that smaller messages achieve a fraction ofpeak bandwidth depending on system scale (e.g., 1 KiB messages achieve 1530% peak bandwidth). Moreover, as system scale grows, larger message sizes
are needed to achieve close-to-peak performance. These observations are notsurprising and apply similarly to most parallel systems. We summarize themquantitatively in Table 4.10. We also chart in detail the effects of message sizeon broadcast bandwidth in Figures 4.9 and 4.10. The charts’ data refer to thesame scales (1-16 IPUs) and experimental topologies (a)-(j) already discussed.
In almost all experimental conditions, bandwidth saturates smoothly, increasing monotonically with block size and suffering no degradation. Minor exceptions are topologies (c) and (d) (broadcasting to a whole IPU acrossboards), where block sizes larger than 64 KiB seem to experience slightly
lower bandwidths than at 64 KiB.
Scale
(IPUs)

Experiment% peakat 1
KiB
messagePeakbandwidth(GiB/s)1... the entire IPU (same as source tile)

30.3 %

13.2

15,1341

to an entire IPU, different than that of source tile... both IPUs on the same board(b)
18.9 %... across boards, direct IPU link(c)
22.1 %... across boards, no direct IPU link(d)16.4 %

5.7
5.4
4.8

6,871
6,433
6,427

to 2 IPUs... both IPUs on the same board... across boards, direct IPU link... across boards, no direct IPU link

(e)
(f)
(g)

25.2 %
31.4 %
29.5 %

7.3
7.5
7.4

9,440
9,051
9,034

to 4 IPUs on two boards
to 8 IPUs on four boards
to 16 IPUs, entire system

(h)
(i)
(j)

16.5 %
15.2 %
14.6 %

22.2
33.6
41.2

25,43736,75647,343

2

4
8
16

(a)

90% peakmessagesize(KiB)

Table 4.10: Summary of aggregate broadcast bandwidth below peak.

52

CHAPTER 4. INTERCONNECT

50(a) on chip(b) across IPUs, on board(c) across boards, direct IPU link(d) across boards, no direct IPU link(e) 2 IPUs, on board(f) 2 IPUs, across boards, direct IPU link(g) 2 IPUs, across boards, no direct IPU link(h) 4 IPUs on 2 boards(i) 8 IPUs on 4 boards(j) 16 IPUs, whole system

45

40

Aggregate Bandwidth (TB/s)

35

30

25

20

15

10

5

0

16

32

64

128

256

512

1k

2k

4k

8k

16k

32k

64k

128k

Block size (B)Figure 4.9: Impact of block size on aggregate broadcast bandwidth. Each line represents a
different experiment topology (a)-(j) as illustrated in Figure 4.7.

4.2. BROADCAST

1453(a) on chip(b) across IPUs, on board(c) across boards, direct IPU link(d) across boards, no direct IPU link(e) 2 IPUs, on board(f) 2 IPUs, across boards, direct IPU link(g) 2 IPUs, across boards, no direct IPU link(h) 4 IPUs on 2 boards(i) 8 IPUs on 4 boards(j) 16 IPUs, whole system

12

Bandwidth per tile (GB/s)

10

8

6

4

2

0

16

32

64

128

256

512

1k

2k

4k

8k

16k

32k

64k 128k

Block size (B)Figure 4.10: Impact of block size on broadcast bandwidth per tile. Each line represents a different
experiment topology (a)-(j) as illustrated in Figure 4.7.

54

CHAPTER 4. INTERCONNECT

IPU systems behave well at scale, with bandwidth growing smoothly withscale as the number IPUs increases to 2, 4, 8, and 16 (experiments (g),(h),(i)and (j) respectively). We see no saturation at scale, and larger scales benefit
from higher aggregate bandwidths. Per-tile bandwidths decrease smoothly,roughly with the logarithm of the system size.

4.3

GatherIn this section we study the latency and bandwidth performance available togather operations.
Gather. For the complete avoidance of doubt, a gather is a collective operationin which one tile receives one message from multiple source tiles. The destination tile receives a distinct message from each source, and must maintaina destination buffer available to the operation, in its local memory, as large
as the product of message size and source count. Each source only needs
to maintain one destination buffer as large as the message size. In benchmarks involving whole IPUs, the source tile is also among the destinations,and must maintain in its local memory one source buffer plus multiple destination buffers.
Scale. We benchmark the same scales as in our broadcast experiments described in Section 4.2.
No load. Experiments in this section study load conditions in which no otheroperation is in progress. Each transfer’s performance is affected only by theload caused by the remainder of the collective operation.
Topologies. We benchmark the topologies discussed in the previous section
and illustrated in Figure 4.7, with the notable difference that data flows arereversed with respect to those of a broadcast operation, depicted in the figureand described in Section 4.2.
Message size. The destination tile must dedicate to the operation one buffer
per source tile in its local memory. This constraints limits the maximum message size usable in a gather operation. In turn, the limited message size is theprimary factor limiting the aggregate bandwidth available to the operation. A
complete discussion follows.
In benchmarks involving whole IPUs, the destination tile is also among
the sources, and maintains at the same time one output buffer and multiple
input buffers in its local memory.

4.3.1

Congestion-free Gather Latency

We study the minimum latency associated with a gather operation. To dothat, we transfer a message of minimum size, i.e., a word of 32 bits. A gather
involving an entire IPU completes in 0.8 microseconds. Gather latencies in4.3. GATHER
Scale

55

Experiment

Source
tile
count

Total
latency(µs)

Avg. latency
per transfer
(ns)

1

0.012

11.557On-chip: source and destination tiles are on the same IPU... from 2 tiles
2
...
4
...
8
...
16
...
38
...
76
...
152
... from a quarter of one IPU
304... from half IPU
608... from the entire IPU
(a)
1,216

0.094
0.094
0.094
0.094
0.097
0.121
0.169
0.264
0.455
0.835

47.14223.567
11.784
5.892
2.563
1.595
1.111
0.869
0.748
0.687

from an entire IPU to a destination tile located on a different IPU... both IPUs on the same board
(b)
1,216
4.929
... across boards, direct IPU link
(c)
1,216
3.205
... across boards, no direct IPU link
(d)
1,216
6.693

4.053
2.636
5.504

Transfer to self (via local memory)

1/8
1/4
1/2
1
1

Multi-IPU experiments
2
IPUs on the same board
2
IPUs on different boards, direct IPU link
2
IPUs on different boards, no direct IPU link
4
IPUs on two boards
8
IPUs on four boards
16
IPUs, entire system

(e)
(f)
(g)
(h)
(i)
(j)

2,432
2,432
2,432
4,864
9,728
19,456

5.708
3.983
7.472
8.995
12.37725.159

2.347
1.638
3.072
1.849
1.272
1.293

Table 4.11: Minimum gather latency in experiments of varying scale and topology. The topology
of experiments (a)-(j) is as illustrated in Figure 4.7, but with reversed data flow directions.

Multi-IPUs of increasing size grow sub-linearly, with 16 IPUs completing awhole-system gather in 25 microseconds. Our results are in Table 4.11.
As we did for the broadcast operation, we first report the latency (12nanoseconds) of a trivial gather operation involving the same tile as sourceand destination. It results in a local memory copy, and we only report it forcomparison and latency breakdown.
Latency off chip. The direct comparison between experiments that involve apair of IPUs, connected directly vs. indirectly by IPU links, reveal that gather
latency depends directly on the number of hops between source and destination IPUs in the ladder network.
Specifically, experiment (d) results in approximately twice as much latency
as experiment (c). Similarly, experiment (g) exhibits roughly twice as muchlatency as (f). This is intuitively consistent with the fact that experiments (g)and (d) involve two hops in the IPU link network, whereas experiments (c)and (f) only involve one hop (see Figure 4.7, but assume the reverse data flow
direction).

56

CHAPTER 4. INTERCONNECT

A gather operation involving two IPUs on the same board exhibits slightly
longer latency when the IPUs are across boards and directly connected. Thisis evident by direct comparison between (c) and (b) in the table and also bycomparison between (f) and (e). This result is consistent with our results forthe broadcast and scatter operations, which we report in Tables 4.8 and 4.13,
respectively.
Latency at scale. Results show that a doubling of system size roughly corresponds to a doubling of network diameter, which causes in turn an approximate doubling of total latency.

4.3.2
Scale

Peak Gather BandwidthExperiment

Sourcetile
countTransfer to self (via local memory)

1/8
1/4
1/2
1
1

Messagesize(bytes)Aggregate
bandwidth(GB/s)

Bandwidth
per transfer(MB/s)

160

0.571

570.61On-chip: source and destination tiles are on the same IPU
To one tile ... from 2 tiles
2... from 4 tiles
4
...
8
...
16
...
38
...
76
...
152
... from a quarter of one IPU
304... from half IPU
608... from the entire IPU
(a)
1,216

160
160
160
160
160
160
160
160
160
160

0.580
1.258
2.105
3.148
4.465
5.260
5.759
6.063
6.219
6.303

290.07314.47263.16
196.75117.51
69.22
37.89
19.94
10.23
5.18

from an entire IPU, different than that of the destination tile... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link
(d)
1,216

80
80
80

4.815
4.805
4.561

3.96
3.95
3.75

80
80
80
40
20
4

5.829
5.685
5.677
5.856
5.743
5.520

2.40
2.34
2.33
1.20
0.59
0.28

Multi-IPU experiments2on the same board
2
across boards, direct IPU link2
across boards, no direct IPU link4
two boards
8four boards
16
eight boards, entire system

1

(e)
(f)
(g)
(h)
(i)
(j)

2,432
2,432
2,432
4,864
9,728
19,456

Table 4.12: Peak gather bandwidth: bandwidth available to gather operations that use the largest
message size allowed by local memory capacity, in varying scales and topologies. We explicitly
report the message used in each experiment, as different scales correspond to different maximum
message sizes. Experiments labeled (a)-(j) use topologies corresponding to these illustrated in
Figure 4.7, but with reversed data flow directions.

In this section we study the peak bandwidth available to gather operations
at different scales and in different topologies. As we did in the previous sections, we study the effects of source-destination IPU proximity in topologies(a)...(j) of Figure 4.7, with the caveat that gather operations involve data flowsof the reverse direction than that depicted in the figure. At each scale we
use the largest message size allowed by local memory capacity. Results showthat the operation’s performance is primarily limited by the small message
size and is less sensitive to scale. Aggregate gather bandwidth degrades very

4.4. SCATTER

57

gracefully with scale, with 16-IPU systems experiencing an aggregate bandwidth that is only 5.3% lower than that available on a 2-IPU board. Our results
are in Table 4.12.
Small messages. Local memory capacity on the source tile is the limiting factor for message size. As scale increases, the maximum usable message size
decreases from 160 to 4 bytes. Small message size limits, in turn, available aggregate bandwidth. The bandwidth decrease is consistent with that alreadybenchmarked in Section 4.2.3.
Off chip. Scatter operations exhibit minimal degradation of performance
when moving from on chip to off chip: aggregate bandwidth decreases from
6.3 to 5.6...5.2 GB/s. The aggregate bandwidth at scales of 2...16 IPUs is virtually identical for gather and scatter operations.

4.4

ScatterIn this section we study the latency and bandwidth performance available toscatter operations.
Scatter. A scatter operation is similar to a broadcast in the sense that both operations involve one source and multiple destination tiles. The two operations
differ in the scatter sending a distinct message to each destination, whereas abroadcast sends the same message to all destination. Moreover, a scatter is thereverse operation of a gather.
Scale. We benchmark the same scales as in our broadcast and gather experiments. We described it in Section 4.2.
No load. Experiments in this section study load conditions in which no otheroperations are in progress. Each transfer’s performance is affected only by theload caused by the remainder of the collective operation.
Topologies. We benchmark the same topologies discussed in the previous
sections, and illustrated in Figure 4.7. Scatter data flow directions are correctly
depicted in the figure (thin blue arrows) and also match those described in
Section 4.2 for broadcast operations.
Message size. The source tile must dedicate to the operation one buffer per
destination in its local memory. This constraints limits the maximum message
size usable in a scatter operation. In turn, the limited message size is theprimary factor limiting the aggregate bandwidth available to the operation.
A complete discussion follows.
In benchmarks involving whole IPUs, the source tile is also among thedestinations, and maintains at the same time one input buffer and multiple
output buffers in its local memory.

58

CHAPTER 4. INTERCONNECT

Scale

Experiment

Destination
tile
count

Total
latency(µs)

Avg. latency
per transfer(ns)Transfer to self (via local memory)

1/8
1/4
1/2
1
1

1

0.012

11.542

On-chip: source and destination tiles are on the same IPU... to 2 tiles
2
...
4
...
8
...
16
...
38
...
76
...
152
... to a quarter of one IPU
304... to half IPU
608... to the entire IPU
(a)
1,216

0.181
0.183
0.186
0.191
0.205
0.226
0.273
0.367
0.555
0.927

90.71345.82323.23711.927
5.384
2.973
1.795
1.207
0.913
0.762

to an entire IPU, different than that of the source tile... both IPUs on the same board
(b)
1,216
... across boards, direct IPU link
(c)
1,216
... across boards, no direct IPU link
(d)
1,216

1.361
1.275
1.404

1.361
1.048
1.155

2.268
2.181
2.181
3.707
7.115
13.729

0.932
0.897
0.897
0.762
0.731
0.706

Multi-IPU experiments
2
IPUs on the same board
2
IPUs on different boards, direct IPU link
2
IPUs on different boards, no direct IPU link
4
IPUs on two boards
8
IPUs on four boards
16
IPUs, entire system

(e)
(f)
(g)
(h)
(i)
(j)

2,432
2,432
2,432
4,864
9,728
19,456

Table 4.13: Minimum scatter latency in experiments of varying scale and topology. The topology
of experiments (a)-(j) is as illustrated in Figure 4.7.

4.4.1

Congestion-free Scatter Latency

We study the minimum latency associated with scatter operations of varying
scale and in different topologies. We transfer a message of minimum size,i.e., a word of 32 bits. We vary the experiment’s scale from one tile to thewhole system. We find that a whole-IPU scatter completes in 0.9 microseconds, only marginally slower than the gather operation of equal scale. Scatter
operations show remarkable scalability, with latencies in Multi-IPUs growing sub-linearly as a function of the number of hops in the ladder network.
A whole-system gather completes in 14 microseconds. Interestingly, off-chip
scatter operations are significantly faster than gathers of equal scale. Quantitative results are in Table 4.11.
As we did in earlier tests, we report first the latency (12 nanoseconds) of atrivial operation involving the same tile as source and destination. It results
in a local memory copy, and its latency is only intended for comparison andcontribution breakdown.
Latency off chip. Results collected over experiments at scale show that scatter
latencies also depend on the number of hops between sources and destination

4.4. SCATTER59

in the ladder network, but the off-chip penalty going from a single-IPU to a
2-IPU test is much smaller for scatter than for gather operations (0.4 vs. 3.1
microseconds).
Comparisons between 2-IPU, direct-vs-indirect topologies yield somewhat
similar considerations as gathers: directly connected pairs of IPUs on different
boards see a marginally better latency pairs of IPUs located on the same board
which, in turn, see a better latency than pairs of IPU not directly connected
via IPU links. The latency differences between the respective topologies are,however, much smaller than those seen in the gather experiments.
At scale. Similarly to what we found for gather operations, a doubling ofsystem size roughly corresponds to a doubling of network diameter, whichcauses in turn an approximate doubling of total latency. Scatter operations
complete, however, almost twice as quickly as gathers of equal scale.

4.4.2
Scale

Peak Scatter BandwidthExperiment

Destination
tile
count

Messagesize(bytes)Aggregate
bandwidth(GB/s)

Bandw.
per tile(MB/s)

1160

13.88413883.50

On-chip: source and destination tiles are on the same IPU
One tile ... to 2 tiles
2... to 4 tiles
4
...
8
...
16
...
38
...
76
...
152
... a quarter of one IPU
304... half IPU
608... the entire IPU
(a)
1,216

160
160
160
160
160
160
160
160
160
160

1.728
2.706
3.812
4.770
5.592
5.971
6.175
6.279
6.332
6.360

864.10676.54476.55298.12147.15
78.56
40.62
20.65
10.41
5.23

to an entire IPU, different than that of source tile... both IPUs on the same board(b)... across boards, direct IPU link(c)... across boards, no direct IPU link
(d)

1,216
1,216
1,216

80
80
80

5.633
5.229
5.217

4.63
4.30
4.29

Multi-IPU experiments
2
IPUs on the same board(e)2
IPUs on different boards - direct IPU link(f)
2
IPUs on different boards - no direct IPU link (g)4IPUs on two boards(h)8
IPUs on four boards(i)16
IPUs, entire system
(j)

2,432
2,432
2,432
4,864
9,728
19,456

80
80
80
40
20
4

5.899
5.685
5.677
5.855
5.743
5.514

2.43
2.34
2.33
1.20
0.59
0.28

Transfer to self (via local memory)

1/8
1/4
1/2
1
1

Table 4.14: Peak scatter bandwidth: bandwidth available to scatter operations that use the largest
message size allowed by local memory capacity, in varying scales and topologies. We explicitly
report the message used in each experiment, as different scales correspond to different maximum message sizes. Experiments labeled (a)-(j) use the corresponding topologies illustrated in
Figure 4.7 and described in Section 4.2.

In this section we study the peak bandwidth available to scatter operations of different scales. As we did in the previous sections, we study theeffect of source-destination IPU proximity in topologies (a)...(j) of Figure 4.7.

60

CHAPTER 4. INTERCONNECT

At each scale we use the largest message size allowed by local memory capacity. Results show that the operation’s performance is primarily limited by
the small message size, and is less sensitive to scale. Aggregate scatter bandwidth degrades very gracefully with scale, with 16-IPU systems experiencing
an aggregate bandwidth that is only 6.6% lower than that available on a 2-IPU
board. Our results are in Table 4.14.
Small messages. We remark that local memory capacity on the source tile isthe limiting factor for message size. As scale increases, the maximum usable
message size decreases from 160 to 4 bytes. Small message size is, in turn,the primary factor for the low aggregate available bandwidth. Results are
consistent with the findings already presented in Section 4.2.3.
Off chip. Scatter operations exhibit minimal degradation of performance
when moving from on chip to off chip: aggregate bandwidth decreases from
6.3 to 5.6...5.2 GB/s. The aggregate bandwidth at scales 2...16 IPUs is virtually
identical for gather and scatter operations.

4.5

All to all

In this section we study the performance of all-to-all collective operations on
the IPU. Because of the number of distinct personalized transfers involvedin this operation and the associated local memory footprint, we were unable
to scale our benchmark to multiple IPUs, or even to an entire IPU. For thatreason, results in this section will be incomplete. We report latency results up
to half IPU (609 tiles). Half an IPU completes an all-to-all in 0.55 microseconds.
Our results are in Table 4.15.
All to all. In an all-to-all operation, each tile in a group sends one distinct
message to each tile in the group, itself included. If the group contains nmembers, each participant concurrently sends n distinct messages. The wholeoperation involves transferring n2 distinct messages.
Message size. Each tile must dedicate to the operation n output buffers andn input buffers. This constraint limits the maximum message size usable in
the operation. Specifically, we only consider messages consisting of a single
word, because they allow us to benchmark the largest scale (1/2 IPU).
Scale. Because of the aforementioned considerations, we were unable tobenchmark a whole-IPU all-to-all operation, or one involving multiple IPUs.

4.6

Reduction

We investigate the performance of reduction operations at different scales andin different topologies.
Reduction. A reduction involves a variable number of source tiles (depending4.6. REDUCTION
Scale

ExperimentTransfer to self (via local memory)

1/8
1/4
1/2
1

On-chip: all tiles are on the same IPU
between 2 tiles
...
...
...
...
...
...
... a quarter of one IPU... half IPU... the entire IPU

61
Total
tile
count

Concurrent
transfers

Total
latency(µs)

Avg. latency
per transfer
(ns)

1

1

0.012

11.542

2
4
8
16
38
76
152
304
608
1,216

4
16
64
256
1,444
5,776
23,104
92,416369,664
1,478,656

0.131
0.125
0.143
0.163
0.182
0.215
0.256
0.355
0.552
-

32.816
7.812
2.227
0.637
0.126
0.037
0.011
0.004
0.001
-

Table 4.15: Minimum all-to-all latency: latency of all-to-all communications in different scales.

on scale) plus one destination tile. The operation takes one input numericalarray on each source tile, and returns one output numerical array on the destination tile. (For the avoidance of doubt, we are describing the equivalent of
an MPI_Reduce primitive, not an MPI_Allreduce.)
Weak vs. Strong Scaling. As we vary the scale of the experiment, we consider
two scenarios: one in which the total amount of input operands grows linearlywith parallelism (weak scaling) and one in which it remains constant (strong
scaling).
Benchmark. The Poplar SDK offers popops::reduce, a library primitive that
performs the reduction. Our experiments simply benchmark this primitive’s
performance. We use sum as a reduction operation.
No load. Experiments in this section study load conditions in which no otheroperation is in progress. Each transfer’s performance is affected only by theload caused by the remainder of the collective operation.
SDK version. Actual performance depends on the library function implementation provided with the specific version of SDK employed. Our results
are only representative of SDK 1.0.49.

4.6.1

Minimum Reduction Latency - Weak Scaling

We study how the latency of one reduction scales (first weakly, then strongly)
as it extends across tiles, then across IPUs on one board, then across boards,
finally spanning an entire 16-IPU system. Interestingly, we find the latency associated with completely distributed reduction is comparable (11...16% more
expensive) than a sequential operation of equal input size entirely performed
on one tile. Detailed results are in Table 4.16.

62

CHAPTER 4. INTERCONNECT

Intent The purpose of this experiment is to understand how larger scale andparallelism affect the overall latency and the per-operand latency of the operation. In this context, we are especially interested in the data transfer component, and not in the arithmetics involved in the reduction. Specifically, thelarger is the scale, the higher is the fraction of messages occurring over slower
interconnects, i.e., IPU link vs. exchange, or more vs. fewer hops.
Input size: in this section, all parallel experiments intentionally use the smallest inputs, i.e., one single-precision floating point value (32 bits) per tile. This
choice exposes primarily the cost of data transfers, as opposed to that of arithmetic computation or local memory access.
For comparison against the parallel experiments, we also perform singletile baseline experiments of equal input size as the parallel ones (i.e., 1,216,
then 2,432, ... finally 19,456 operands). These measurements serve as a baseline for the cost of a reduction when no parallelism is involved.

4.6. REDUCTION

Experiment

63

Operands
per
tile

Tiles
involved

Total
latency(µs)

Avg. latency
per input
operand (ns)

1,216
2,432
4,864
9,728
19,456

1
1
1
1
1

1.69
2.44
3.97
7.02
13.10

1.39
1.00
0.82
0.72
0.67

Increasing system diameter
1 IPU, 1 tile
2 IPUs on the same board
4 IPUs on two PCI boards
8 IPUs on four PCI boards
16 IPUs, entire system

1
1
1
1
1

1
2
4
8
16

0.44
1.13
1.48
2.02
3.16

0.44
0.57
0.37
0.25
0.20

Increasing scale
1 IPU, all 1,216 tiles
2 IPUs on the same board
2 IPUs on different boards, direct IPU link
2 IPUs on different boards, no direct IPU link
4 IPUs on two PCI boards
8 IPUs on four PCI boards
16 IPUs, entire system

1
1
1
1
1
1
1

1,216
2,432
2,432
2,432
4,864
9,728
19,456

1.97
5.32
3.60
7.09
7.18
7.68
14.51

1.62
2.19
1.48
2.92
1.48
0.79
0.74

Baseline sequential
1 tile1 tile1 tile1 tile1 tileTable 4.16: Reduction, weak scaling: Latency of one reduction as it scales up from a single tile
to an entire 16-IPU system.

64

CHAPTER 4. INTERCONNECT

ExperimentBaseline - one tile
1 IPU, all 1,216 tiles
2 IPUs on the same board
2 IPUs on different boards, direct IPU link
2 IPUs on different boards, no direct IPU link
4 IPUs on two PCI boards
8 IPUs on four PCI boards
16 IPUs, entire system

Operands
per
tile

Tiles
involved

Total
latency(µs)

Avg. latency
per input
operand (ns)19,456

1

13.1

0.67

16
8
8
8
4
2
1

1,216
2,432
2,432
2,432
4,864
9,728
19,456

2.1
5.4
3.7
7.20
7.28
7.78
14.51

0.11
0.28
0.19
0.37
0.37
0.40
0.75

Table 4.17: Reduction, strong scaling: Latency of one reduction as it scales up from a single tile
to an entire 16-IPU system.

4.6.2

Minimum Reduction Latency - Strong Scaling

We benchmark reduction latency once more, but in a strong scaling scenario:
problem size is kept constant across experiments at different scales, in order
to expose the cost of parallelization. We choose the smallest input size thatwould place one operand per tile on the largest system considered, i.e., 19,456
operands. Reducing 19,456 single-precision values on one IPU takes 2.1 microsecond. Performing the same reduction on a 16-IPU system costs 14.51
microseconds.

4.6.3

Peak Reduction Bandwidth

We measure the throughput achieved by reductions of different sizes andscale. As a measure of reduction throughput, we adopt the input operandsize consumed in the unit of time. In this section, all experiments study weakscaling. For simplicity, we report quantitative results for only two choices(1,000 and 25,000 operands per tile) of input size, in Table 4.18. In Figure 4.11
we chart the effect of operand size on reduction bandwidth in a broader andmore detailed spectrum of input sizes.
On-chip scaling. In weak scaling conditions, on-chip aggregate reduction
bandwidth grows remarkably well with system size. The aggregate per-chip
throughput reaches 6.76 TB/s, with a parallel efficiency equal to 92.2%.
Off-chip scaling. For larger inputs (25,000 values per tile), the aggregate
throughput grows by a factor of 9.4× as the system scales up from 1 to 16
IPUs. The peak aggregate throughput of a 16-IPU system reaches an impressive 64.7 TB/s. The per-tile bandwidth degradation is approximately 41%; theparallel efficiency is 59%.
Operand size. The effect of operand size on performance is evident by direct comparison between the third and fourth band of the table, which report results of experiments with 1,000 and 25,000 values per tiles, respectively.

4.6. REDUCTION
Experiment65
Operands
per
tile

Tiles
involved

Aggregate
bandwidth(GB/s)

Bandw.
per tile(GB/s)

Single-IPU baseline performance
Baseline - 1,000 tiles
Baseline - 1,000 tiles
Baseline - 1,000 tiles
Baseline - 1,000 tiles
Baseline - 1,000 tiles

1,216
2,432
4,864
9,728
19,456

1,000
1,000
1,000
1,000
1,000

1,671
2,657
3,747
4,729
5,432

1.67
2.68
3.75
4.73
5.43

Large operands - Single-IPU Weak scaling
Running on ... one tile... 2 tiles... 4 tiles
...
...
...
...
...
... a quarter of one IPU... half IPU... the entire IPU

25,00025,00025,00025,00025,00025,00025,00025,00025,00025,00025,000

1
2
4
8
16
38
76
152
304
608
1,216

6
12
24
47
92
224
448
890
1,769
3,481
6,756

6.03
5.97
5.91
5.86
5.72
5.90
5.89
5.85
5.82
5.73
5.56

Smaller operands - Multi-IPU Weak scaling
1 IPU, all 1,216 tiles
2 IPUs on the same board
2 IPUs on different boards, direct IPU link
2 IPUs on different boards, no direct IPU link
4 IPUs on two PCI boards
8 IPUs on two PCI boards
16 IPUs, entire system

1,000
1,000
1,000
1,000
1,000
1,000
1,000

1,216
2,432
2,432
2,432
4,864
9,728
19,456

1,636
1,540
2,090
1,230
2,381
4,512
5,016

1.34
0.63
0.86
0.51
0.49
0.46
0.26

Larger operands - Multi-IPU Weak scaling
1 IPU, all 1,216 tiles
2 IPUs on the same board
2 IPUs on different boards, direct IPU link
2 IPUs on different boards, no direct IPU link
4 IPUs on two PCI boards
8 IPUs on two PCI boards
16 IPUs, entire system

25,00025,00025,00025,00025,00025,00025,000

1,216
2,432
2,432
2,432
4,864
9,728
19,456

6,756
11,39912,400
10,48520,965
41,12863,724

5.56
4.69
2.55
2.16
4.31
4.23
3.28

Table 4.18: Reduction bandwidth available at different scales (sequential, 1-16 IPUs) and for
different input sizes, in weak scaling conditions.

Larger operands achieve a substantially higher bandwidth at any scale, andbetter scalability overall: the large-vs-small operand speedup is 4.1× on a single IPU, but it grows to 12.7× on a 16-IPU system. We chart a broader set ofresults describing the same trends in Figure 4.11.

66

CHAPTER 4. INTERCONNECT

80(a) Reduction on IPU [0]75(b) Reduction on IPU [0,1](c) Reduction on IPU [0,2]70(d) Reduction on IPU [0,3](e) Reduction on IPU [0-3]65(f) Reduction on IPU [0-7]60(g) Reduction on IPU [0-15]

Aggregate Bandwidth (TB/s)

55
50
45
40
35
30
25
20
15
10
5
0

16

32

64

128

256

512

1k

2k

4k

8k

16k

32k

Operand count per tile

Figure 4.11: Effect of operand size on reduction bandwidth in weak scaling conditions. IPU
identifiers are DNC IDs.

4.7. HOST CONNECTIVITY

4.7

67

Host Connectivity

In this section we present bandwidth and latency associated with host-to-IPU
data transfers. These metrics are relevant to evaluate potential bottleneck inachieving high performance in hybrid CPU/IPU applications where the PCIexpress bus lies in the critical path. Bandwidth is important for streaming
applications where CPU data are continuously offloaded to the IPU, and latency is important for inference applications where the output is produced
in a ping-pong fashion across CPU and IPU. The reader should pay consideration to fact that PCI-express is an industry-standard I/O technology; theresults presented here are meant to be compared against the theoretical limit.
SDK version. Performance depends on the version of SDK and drivers employed. Our results are only representative of SDK 1.0.49. We expect Graphcore to further optimize host-to-IPU connectivity performance in future releases.

4.7.1

Minimum Host-to-IPU Latency

Benchmark description: we use a benchmark that sends data from the hostto a tensor on IPUs using the graph::DataStream mechanism with an empty
callback to minimize overhead associated with the software stack. Depending on the benchmark scale and IPU count, the tensor is distributed acrosslocal memories such as each tile receives exactly 4 byte data. The latency
measurements are taken on the host side as average over multiple multiple
synchronous transfers.
Observations: as shown in Table 4.19, the latency to transfer data to an IPU
is at least 8.81 µs and is stable across the PCI-express topology. Sending
data to different destination IPUs doesn’t show obvious differences in latency.
Data transfer latency also doesn’t increase when communicating with multiple IPUs. We only notice a slight increase in latency as destination tile countincreases. This, however, is a consequence of moving a tensor large enough to
fit the maximum PCI-express payload size, requiring multiple transactions tocomplete the transfer.

4.7.2

Peak Host-to-IPU Bandwidth

Benchmark description: we measure aggregate bandwidth with a benchmarkthat transfers data from the host to a tensor on IPUs via graph::DataStream.
The destination tensor is partitioned linearly across all involved tiles and
IPUs. The benchmark sends 40 KB data to each tile, and up to 778.24 MB
to all 16 IPUs. We report results in Table 4.20.
Observations: the achieved aggregate bandwidth increases with the growthof involved PCIe lane count. Since each IPU is connected to the PCIe switch
via 8 PCIe Gen3 lanes, transfers to one IPU reaches up to ∼ 6 GB/s. Every

68

CHAPTER 4. INTERCONNECT

Scale(IPUs)

1/8
1/4
1/2
1
1

2
2
2
4
8
16

Experiment

Destination tiles are on IPU 0
From CPU ... to 1 tile... to 2 tiles
...
...
...
...
...
...
... to a quarter of one IPU... to half IPU... to the entire IPU

Destination
tiles

Total
latency
(µs)

1
2
4
8
16
38
76
152
304
608
1,216

8.81
8.84
8.89
8.31
8.81
8.81
9.01
8.80
8.95
9.56
9.87

Single destination IPU... to 1 tile on IPU 0... to 1 tile on IPU 1... to 1 tile on IPU 2... to 1 tile on IPU 3... to 1 tile on IPU 4... to 1 tile on IPU 5... to 1 tile on IPU 6... to 1 tile on IPU 7... to 1 tile on IPU 8... to 1 tile on IPU 9... to 1 tile on IPU 10... to 1 tile on IPU 11... to 1 tile on IPU 12... to 1 tile on IPU 13... to 1 tile on IPU 14... to 1 tile on IPU 15

1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

8.87
8.89
8.83
8.80
8.82
8.81
8.83
8.85
8.84
8.80
8.82
8.81
8.80
8.81
8.88
8.82

Multiple destination IPUs, 1 tile each IPU... to 2 IPUs on the same board... to 2 IPUs across boards, direct IPU link... to 2 IPUs across boards, no direct IPU link... to 4 IPUs on two boards... to 8 IPUs on four boards... to 16 IPUs, entire system

2
2
2
4
8
16

8.88
8.83
8.84
8.83
8.82
8.83

Table 4.19: Minimum host-to-IPU latency in experiments of different scale and topology.

4.7. HOST CONNECTIVITY
Scale(IPUs)

Experiment69
Destination
tiles

PCIe
lanes

Aggregate
bandwidth(GB/s)

Bandw.
per tile(MB/s)

...
...
...
...
... to a quarter of one IPU... to half IPU... to the entire IPU

1
2
4
8
16
38
76
152
304
608
1,216

8
8
8
8
8
8
8
8
8
8
8

2.57
3.54
4.41
5.07
5.43
5.85
5.74
5.80
5.82
5.87
5.86

2,565.29
1,768.85
1,103.05
634.30339.50154.04
75.50
38.17
19.15
9.66
4.82

Single destination IPU... to all tiles on IPU 1... to all tiles on IPU 2... to all tiles on IPU 3... to all tiles on IPU 4... to all tiles on IPU 5... to all tiles on IPU 6... to all tiles on IPU 7... to all tiles on IPU 8... to all tiles on IPU 9... to all tiles on IPU 10... to all tiles on IPU 11... to all tiles on IPU 12... to all tiles on IPU 13... to all tiles on IPU 14... to all tiles on IPU 15

1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216
1,216

8
8
8
8
8
8
8
8
8
8
8
8
8
8
8

5.84
5.85
5.87
5.84
5.84
5.84
5.84
5.97
6.02
6.03
6.02
6.03
6.03
6.03
6.03

4.80
4.81
4.82
4.80
4.80
4.80
4.80
4.91
4.95
4.96
4.95
4.96
4.96
4.96
4.96

Multiple destination IPUs... to 2 IPUs on the same board... to 2 IPUs across boards, direct IPU link... to 2 IPUs across boards, no direct IPU link... to 4 IPUs on two boards... to 8 IPUs on four boards... to 16 IPUs, entire system

2,432
2,432
2,432
4,864
9,728
19,456

16
16
16
16
32
64

11.35
11.36
11.35
13.78
27.55
55.04

4.67
4.67
4.67
2.83
2.83
2.83

Destination tiles are on IPU 0
From CPU to 1 tile... to 2 tile

1/8
1/4
1/2
1
1

2
2
2
4
8
16

Table 4.20: Peak host-to-IPU bandwidth available to concurrent transfers in different transfer
topologies.

four IPUs share the same 16 PCIe lanes, thus the transfer to 4 IPUs (h) canonly achieve 13.78 GB/s. The transfer to all 16 IPUs (j) enjoys a bandwidth of
55.04 GB/s.

Chapter 5

Notable Arithmetic Primitives
5.1

Matrix Multiplication

Dense matrix multiplication (matmul) is a workload of such pervasive presence in HPC and AI/ML applications that its performance on a computing architecture is frequently used (and sometimes abused) as a singular proxy for
an architecture’s overall performance. In this section, we report and discuss
IPU dense matrix multiplication performance as offered by poplin, Graphcore’s linear algebra library, at the time of writing. We compare the IPU’s matmul performance with contemporary GPUs in terms of aggregate throughout
and energy efficiency. We find that the IPU offers an impressive arithmetic
throughput, up to 31.1 TFlops/s in single precision and 124.5 TFlops/s inmixed precision per chip, surpassing the GPU’s theoretical limits. Actual performance measured on GEMM benchmarks show the IPU as a clear winnerin single precision against NVidia’s V100 GPU. In mixed precision, the comparison does not yield a clear winner and requires a more nuanced discussionthat follows.
IPU-GPU comparisons. The IPU offers impressive theoretical compute power
thanks to its specialized hardware called Accumulating Matrix Product (AMP)
units. These units are similar in purpose to the GPU’s TensorCore units. In
the comparisons that follow, the reader should pay consideration to the factthat while one C2 IPU board contains two IPU processors, a Volta GPU board(either PCI or SXM2) only contains one GPU processor; we focus on the perchip comparisons as of interest to architecture designers.
Theoretical limits. Each IPU tile contains one AMP unit. An AMP unit canfinish 64 mixed-precision or 16 single-precision floating point operations per
clock cycle. At a 1.6-GHz clock rate, the 1,216 tiles on one IPU deliver 31.1
and 124.5 TFlops/s in single and mixed precision, respectively. In single precision, one IPU processor offers almost twice as much single-precision theoretical throughput as one V100 GPU: 31.1 vs. 15.7 TFlops/s. This result also
reflects the fact that on the GPU, TensorCores do not support single precision,71

72

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES
Arithmetic

Graphcore C2 IPU

NVidia Volta V100 (SXM2)Precision

Units

1 IPU

Units

1 GPU

Theoretical
Theoretical

AMP
Vector

31.1
7.8

TensorCores
FP cores

N/A
15.7

Actual GEMM
% Theor.

AMP

18.9
60.7%

FP cores

15.5
98.7%

Theoretical
Theoretical

AMP
Vector

124.5
15.6

TensorCores
FP cores

125.0
31.4

Actual GEMM
% Theor.

AMP

58.9
47.3%

TensorCores

90.0
72.0%

Single(FP32)

Mixed(FP16.32)

Table 5.1: Arithmetic throughput per-chip comparison between IPUs and GPUs. We offer theoretical upper bounds and actual peak performance that we benchmarked on matrix-matrix multiplication (GEMM). Both platforms offer specialized units; we offer independent theoretical upperbounds for specialized and non-specialized units on both platforms. Benchmarks used the respective vendors’ optimized GEMM functions.

and single-precision computation uses regular FP cores. In mixed precision,
one IPU processor roughly matches one V100 GPU (124.5 vs. 125.0 TFlops/s).
Rows labeled as “Theoretical” in Table 5.1 report these numbers side to side,
offering separate limits for specialized and non-specialized units.
Benchmark. Our experiments benchmark each device’s performance as all of
the chip’s resources are used to perform one, large, matrix-matrix multiplication. We study performance sensitivity to input size. For simplicity, we only
consider square input matrices, i.e., matmul of A and B, both of size n × n.
We vary n from 16 to the largest size fitting matrices.
Version sensitivity. Because we are benchmarking library functions, performance is also a factor of code optimization, not just hardware compute power.
We report actual GEMM performance delivered on each device by the respective manufacturers’ optimized linear algebra libraries: Graphcore’s poplin
and NVidia’s cuBlas. Our IPU results only describe SDK version 1.0.49, andmay differ significantly from future, more optimized poplin versions. Sameconsiderations apply to the GPU, where we used cuBLAS version 10.1.0.105.
Single precision. We measured 18.9 TFlops/s per IPU at peak, which is 60%of the theoretical limit. The two IPUs on a C2 board can deliver twice asmuch performance (37.8 TFlops/s) if used in parallel on independent matrix
operands. The IPU outperforms the GPU in a per-chip comparison. This result reflects the fact that the GPU’s specialized TensorCore units do not support pure single precision. In Table 5.1, “Actual GEMM” rows report theseresults, also specifying what hardware units the library’s GEMM implementation uses at each precision and what fraction of theoretical throughput they
achieve.

5.1. MATRIX MULTIPLICATION

73
Precision

Theoretical arithmetic throughput
Actual, Peak
Actual/theoretical fraction

SingleMixed

31.1 TFlops/s
18.9 TFlops/s
60.7 %

124.5 TFlops/s
58.2 TFlops/s
46.7%

Table 5.2: Theoretical arithmetic throughput compared to the actual peak throughput we measured on dense matrix multiplication on one IPU.

While the IPU delivers higher throughput, the GPU supports larger
operands thanks to its higher device memory capacity. In our experiments,
the largest square matrix operands fitting one IPU are 2,944×2,944, while on a
32-GB GPU they are roughly ∼50,000× ∼50,000.
Mixed precision. On both devices, specialized hardware (TensorCores and
AMP units) supports matrix multiplication in mixed precision. Despite one
IPU delivering roughly the same theoretical throughput as one GPU, in
GEMM benchmarks the IPU yields lower performance than a V100 GPU: 58.9
TFlops/s vs. 90.0 TFlops/s, respectively. The IPU uses a lower fraction of its
theoretical limit (47.3%) than the GPU (72.0%). See the lower half of Table 5.1.
In mixed precision as well, the GPU supports larger operands. The largest
square matrix operands fitting an IPU are 2,688×2,688, while on a 32-GB GPU
they are roughly ∼72,000× ∼72,000.
Tile mapping. We benchmark two mappings of input operands to tile memory:• basic: this benchmark invokes function poplin::matMulAcc, mapping matrices to tiles linearly via function poputil::mapTensorLinearly.
Specifically, input matrices are spreads evenly over tiles in a linear lexicographic manner, with the indices of the flattened matrix mapped
across increasing tile IDs;
• optimized: this benchmark includes an optimization that preconditions
the two input operands via functions poplin::createMatMulInputLHS and...RHS. It then uses poplin::matMul and popops::scaledAddTo.
Comparison with GPUs. We compare IPUs and GPUs in terms of throughput
and energy efficiency in single and mixed precision. This is a measurement of
how well a platform can minimize the latency to complete matmuls when theentirety of the processor is available for a single task. We offer comparisonson a per-chip and per-board basis. Since the IPU board has two chips, we consider a library-based matmul benchmark that extends the matmul operation

74

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES

Arithmetic throughput (TFlops/s)

Single precision
20
15
10
5
Basic
Opt-mapping

0
0

500

1000

1500

2000

Matrix size

Arithmetic throughput (TFlops/s)

Mixed precision
60
50
40
30
20
10

Basic
Opt-mapping

0
0

500

1000

1500

2000

Matrix size
Figure 5.1: Floating-point arithmetic throughput achieved by dense matrix multiplication on one
IPU. For simplicity, we use square inputs (n × n) × (n × n); the horizontal axis varies n.

to both IPUs on a C2 board. The GPU we consider is NVidia’s V100. Table 5.2
summarizes our findings.
We chart results in Figure 5.1, where the horizontal axis represents n, oneside of each square input operand matrix. Performance varies with input size,with smaller matrices unable to achieve sufficient occupancy of the device.

5.2. CONVOLUTION

75This is a common occurrence among massively parallel computing platform
and is of little consequence in most cases. In our benchmark using optimized
tile mapping, performance saturates around 18 and 55 ... 58 TFlops/s in single
and mixed precision, respectively.
We also compare the devices in terms of energy efficiency by charting
throughput values divided by nominal power absorbed by each board in Figure 5.3. Results are expressed in units of TFlops/s per Watt (TFlops/s/W).
For the V100 GPU, we use nominal power, 250 W.
In both single and mixed precision cases, a single IPU delivers higher efficiency than two IPUs and higher efficiency than the V100 GPU.

5.2

Convolution

We dedicate this section to studying the IPU’s convolution performance. For
our benchmarking suite, we selected a basket of 6 commonly used CNNs
in image classification according to our survey of recent literature [9, 10, 11,
12, 13, 14]. We compare the IPU’s and the V100 GPU’s forward-pass performances, on a per-chip basis, first in terms of arithmetic throughput (TFlops/s),
and then in terms of throughput normalized per nominal power consumption(TFlops/s/W). In our results, IPUs tend to outperform GPUs at smaller batch
sizes. While the GPU supports larger batches thanks to its larger device memory, it also needs to use larger batches for its compute resources to achievesufficient occupancy. For CNNs whose architectures were designed without
GPU efficiency in mind (such as ResNeXt), we observe speedups upward of
700x.
Benchmarking. On each device, we use the manufacturer’s respective optimized convolution primitives: poplin::convolution from the Poplar SDK version 1.0.49, and cudnnConvolutionForward from cuDNN 7.5.0. All benchmarks
use half precision floating point math.
Batch size. On both platforms, we vary batch size over a meaningful range.
Note that sufficiently large batches exceed the memory capacity of both IPUs
and GPUs. Because of the larger GPU device capacity, it supports larger batch
sizes for most layers. Our experiments sweep the range 1 ... 2,048 on the GPU
and 1 ... 128 on the IPU.
Raw results. In Tables 5.4 and 5.5 we report in detail the arithmetic throughput achieved by each layer of each CNN, for each batch size considered, on
the GPU and IPU respectively. Results show that the GPU prefers larger batch
sizes, with most network layers reaching their peak performances at batch
size 512 or 1,024. In contrast, the IPU supports smaller batch sizes and tends
to achieve peak performance with batch sizes in the 8...32 range. Moreover,for sizes below peak, the IPU still delivers a good fraction of peak throughput.
Comparison at peak. We now present a simpler and more concise analy-

76

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVESSingle precision

Arithmetic throughput (TFlops/s)

30
25
20
15
10
5
0

0

500

1000

1500

2000

2500

3000

3500

4000

3000

3500

4000

Matrix size
IPUx1

IPUx2

V100

Mixed precision

Arithmetic throughput (TFlops/s)

100
80
60
40
20
0

0

500

1000

1500

2000

2500

Matrix size
IPUx1

IPUx2

V100

Figure 5.2: Matrix multiplication performance on IPUs and GPUs.

5.2. CONVOLUTION

77

Single precision

Energy efficiency (TFlops/s/W)

0.14
0.12
0.1
0.08
0.06
0.04
0.02
0

0

500

1000

1500

2000

2500

3000

3500

4000

3000

3500

4000

Matrix size
IPUx1

IPUx2

V100

Energy efficiency (TFlops/s/W)

Mixed precision
0.6
0.5
0.4
0.3
0.2
0.1
0

0

500

1000

1500

2000

2500

Matrix size
IPUx1

IPUx2

V100

Figure 5.3: Energy efficiency while running matrix multiplication on IPUs and GPUs.

78

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES

sis that abstracts away from batch size. Specifically, we compare the IPU’sand the GPU’s peak performance at each CNN layer (at the respective peak,whichever batch sizes they correspond to). Table 5.3 shows both arithmetic
throughput comparisons and energy efficiency comparisons. We use nominal
power figures (250 W for the GPU, 150 W for the IPU). Columns labeled Ratio
show the IPU/GPU throughput and energy efficiency ratios, with over-unity
values indicating an IPU advantage.
Discussion. Results in Table 5.3 support the expectation that the highly parallel fine-grained MIMD architecture of the IPU would provide an advantage inmodels using group or separable convolutions. When G=32 for the ResNeXt
model, the IPU shows >100x advantage for certain convolutions. This advantage is not available in legacy CNN models that were originally optimized for
the GPU architecture.
Table 5.3: Comparison between one V100 GPU and one IPU (per chip) in terms of arithmetic
throughput and energy efficiency (i.e., arithmetic throughput normalized over nominal power).
Benchmark: reference basket of convolutional neural networks [9, 10, 11, 12, 13, 14]. The ratio
column reports IPU-over-GPU ratios; over-unity values mean that the IPU is performing better
than the GPU; below-unity, vice versa. Parameters: Fs : filter size (height = width); Is : input size
(height = width); Ic : input channel; Oc : output channel; S: stride; G: group.

Layer

Arithmetic Throughput(TFlops/s)

Parameters

Energy Efficiency(TFlops/s/W)

Fs

Is

Ic

Oc

S

G

IPU

V100

Ratio

IPU

V100

Ratio

7
3
1
3
1
1
3
1
1
3
1
1
3
1

224
112
56
56
56
28
28
28
14
14
14
7
7
7

3
64
64
128
128
256
256
256
512
512
512
1024
1024
1024

64
64
128
128
256
256
256
512
512
512
1024
1024
1024
2048

2
2
1
1
1
1
1
1
1
1
1
1
1
1

1
1
32
32
32
32
32
32
32
32
32
32
32
32

15.04
62.07
0.32
2.58
1.14
6.90
36.13
10.22
28.21
45.69
31.57
28.05
37.14
33.41

6.26
22.85
0.04
0.44
0.15
0.02
0.75
0.03
0.04
1.38
0.06
0.08
4.74
0.11

2.40
2.72
8.11
5.86
7.59
344.9648.17
340.59705.2333.11
526.23350.617.84
303.76

0.10
0.41
0.00
0.02
0.01
0.05
0.24
0.07
0.19
0.30
0.21
0.19
0.25
0.22

0.03
0.09
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.02
0.00

4.00
4.53
13.51
9.77
12.65
574.9480.29
567.661175.3855.18
877.05584.3513.06
506.27

ResNet-50 v1.5
Conv1
7
3
Conv2 1
Conv2 2
1
Conv2 3
3
Conv2 4
1
Conv3 1
1
Conv3 2
3
Conv3 3
1
Conv4 1
1
Conv4 2
3
Conv4 3
1
Conv5 1
1
Conv5 2
3
Conv5 3
1

224
112
56
56
56
28
28
28
14
14
14
7
7
7

3
64
64
64
64
256
128
128
256
256
256
512
512
512

64
64
64
64
256
128
128
512
256
256
1024
512
512
2048

2
2
1
1
1
1
1
1
1
1
1
1
1
1

1
1
1
1
1
1
1
1
1
1
1
1
1
1

15.04
62.32
52.39
73.43
58.55
54.87
58.13
62.39
44.41
52.78
60.92
41.43
52.71
57.72

6.13
23.27
8.48
47.74
12.09
20.71
64.07
21.38
26.61
78.29
33.67
37.73
85.48
48.69

2.45
2.68
6.18
1.54
4.84
2.65
0.91
2.92
1.67
0.67
1.81
1.10
0.62
1.19

0.10
0.42
0.35
0.49
0.39
0.37
0.39
0.42
0.30
0.35
0.41
0.28
0.35
0.38

0.02
0.09
0.03
0.19
0.05
0.08
0.26
0.09
0.11
0.31
0.13
0.15
0.34
0.19

4.09
4.46
10.30
2.56
8.07
4.42
1.51
4.86
2.78
1.12
3.02
1.83
1.03
1.98

Inception v3
Conv1

299

3

32

2

1

7.08

3.91

1.81

0.05

0.02

3.02

ResNeXt
Conv1
Conv2 1
Conv2 2
Conv2 3
Conv2 4
Conv3 1
Conv3 2
Conv3 3
Conv4 1
Conv4 2
Conv4 3
Conv5 1
Conv5 2
Conv5 33

5.2. CONVOLUTION

Layer

79
Arithmetic Throughput(TFlops/s)

Parameters

Energy Efficiency(TFlops/s/W)

Fs

Is

Ic

Oc

S

G

IPU

V100

Ratio

IPU

V100

Ratio

Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8
Conv9
Conv10

3
3
3
3
3
3
3
3
3

149
147
147
73
71
35
35
17
8

32
32
64
64
80
192
288
768
1280

32
64
64
80
192
288
768
1280
2048

1
1
2
1
2
1
2
2
1

1
1
1
1
1
1
1
1
1

71.59
70.25
57.36
65.66
61.90
62.12
50.85
45.56
49.27

24.05
38.33
23.39
37.49
32.77
61.78
59.17
72.17
87.92

2.98
1.83
2.45
1.75
1.89
1.01
0.86
0.63
0.56

0.48
0.47
0.38
0.44
0.41
0.41
0.34
0.30
0.33

0.10
0.15
0.09
0.15
0.13
0.25
0.24
0.29
0.35

4.96
3.05
4.09
2.92
3.15
1.68
1.43
1.05
0.93

VGG16
Conv1 1
Conv1 2
Conv2 1
Conv2 2
Conv3 1
Conv3 2
Conv4 1
Conv4 2
Conv5 1

3
3
3
3
3
3
3
3
3

224
224
112
112
56
56
28
28
14

3
64
128
128
256
256
512
512
512

64
128
128
256
256
512
512
512
512

1
1
1
1
1
1
1
1
1

1
1
1
1
1
1
1
1
1

7.20
75.04
72.60
71.79
70.20
72.10
58.53
58.54
49.92

3.83
52.01
64.59
70.46
78.75
81.35
87.22
87.28
87.02

1.88
1.44
1.12
1.02
0.89
0.89
0.67
0.67
0.57

0.05
0.50
0.48
0.48
0.47
0.48
0.39
0.39
0.33

0.02
0.21
0.26
0.28
0.32
0.33
0.35
0.35
0.35

3.13
2.40
1.87
1.70
1.49
1.48
1.12
1.12
0.96

SSD v1.1
Conv1
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8
Conv9
Conv10

3
1
1
3
1
3
1
3
1
3

38
19
19
10
10
10
5
5
3
3

512
1024
1024
512
512
128
256
128
256
128

1024
1024
512
512
128
256
128
256
128
256

2
1
1
2
1
2
1
1
1
1

1
1
1
1
1
1
1
1
1
1

53.35
54.83
57.01
46.76
35.19
31.83
16.21
30.16
8.81
17.56

71.51
53.46
47.1
63.02
24.09
52.52
16.07
63.86
10.12
45.51

0.75
1.03
1.21
0.74
1.46
0.61
1.01
0.47
0.87
0.39

0.36
0.37
0.38
0.31
0.23
0.21
0.11
0.20
0.06
0.12

0.29
0.21
0.19
0.25
0.10
0.21
0.06
0.26
0.04
0.18

1.24
1.71
2.02
1.24
2.43
1.01
1.68
0.79
1.45
0.64

AlexNetConv1
MaxPool1
Conv2
MaxPool2
Conv3
Conv4
Conv5

11
3
5
3
3
3
3

227
55
27
27
13
13
13

3
96
96
256
256
384
256

96
96
256
256
384
256
256

4
2
1
2
1
1
2

1
1
1
1
1
1
1

20.04
51.73
64.34
49.21
53.70
52.91
41.77

14.7
33.12
86.46
50.37
82.07
82.02
54.93

1.36
1.56
0.74
0.98
0.65
0.65
0.76

0.13
0.34
0.43
0.33
0.36
0.35
0.28

0.06
0.13
0.35
0.20
0.33
0.33
0.22

2.27
2.60
1.24
1.63
1.09
1.08
1.27

Table 5.4: Convolution performance (arithmetic throughput, TFlops/s) of one NVidia V100 GPU,in half precision, over our reference basket of convolutional neural network architectures. Missing
data represents experiments that did not fit in device memory.

Layer name
ResNeXt
Conv1
Conv2 1
Conv2 2
Conv2 3
Conv2 4
Conv3 1
Conv3 2
Conv3 3
Conv4 1
Conv4 2

1

2

4

8

16

2.71
5.89
0.00
0.11
0.01
0.00
0.10
0.01
0.00
0.10

3.75
9.92
0.01
0.20
0.03
0.01
0.20
0.02
0.01
0.20

4.05
15.04
0.01
0.32
0.05
0.02
0.39
0.02
0.02
0.36

5.01
17.91
0.03
0.40
0.11
0.02
0.75
0.02
0.04
0.73

5.89
18.42
0.04
0.43
0.14
0.02
0.17
0.03
0.04
1.38

Batch size
32
64
6.02
21.02
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.34

6.10
22.25
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.36

128

256

512

1024

2048

6.14
22.68
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.35

6.21
22.85
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.36

6.24
22.79
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.35

6.26
22.72
0.04
0.44
0.15
0.02
0.17
0.03
0.04
0.36

6.25
22.63
0.04
0.44
0.15
0.02
0.18
0.03
0.04
0.35

80

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES

Layer name

Batch size
32
64

128

256

512

1024

2048

0.05
0.07
1.44
0.09

0.05
0.07
2.80
0.10

0.06
0.08
0.70
0.11

0.06
0.08
4.74
0.11

0.05
0.08
0.72
0.11

0.06
0.08
0.72
0.11

0.06
0.08
0.73
0.11

0.06
0.08
0.73
0.11

4.94
18.15
6.40
29.06
9.76
12.23
28.79
14.69
7.94
18.04
21.85
6.10
8.35
21.96

5.78
18.70
6.30
32.83
10.65
15.95
40.58
17.34
13.61
34.50
23.42
11.46
16.80
31.54

5.91
21.38
7.12
37.18
11.22
16.02
42.01
19.04
18.80
48.05
28.51
20.43
32.97
33.78

5.98
22.72
7.79
42.50
11.69
18.36
53.38
19.88
19.62
50.21
32.23
28.45
51.04
42.23

6.02
23.13
8.15
44.11
11.91
19.93
61.81
20.59
22.74
65.12
31.85
29.07
54.24
48.69

6.09
23.27
8.32
45.32
12.01
20.33
62.70
20.97
25.24
76.76
32.38
34.73
70.48
47.44

6.13
23.23
8.42
46.58
12.07
20.53
63.32
21.21
25.85
77.85
32.94
36.68
82.64
43.98

6.13
23.14
8.46
47.41
12.08
20.61
63.44
21.33
26.22
78.10
33.36
37.27
84.69
44.33

6.13
23.07
8.48
47.74
12.09
20.71
64.07
21.38
26.61
78.29
33.67
37.73
85.48
44.76

1.15
16.71
26.73
17.35
20.81
19.25
40.86
28.60
14.62
16.04

1.30
19.13
30.21
18.12
26.57
21.16
43.01
53.40
24.16
32.20

1.42
21.09
33.03
20.40
31.35
24.51
56.34
57.41
47.03
55.52

1.46
22.43
34.69
22.21
34.52
28.09
57.85
59.17
52.68
65.78

1.50
23.29
36.28
23.21
36.46
30.42
58.77
52.52
71.49
72.01

1.52
23.67
37.27
23.28
36.68
31.59
61.21
55.04
72.17
84.09

1.53
23.89
37.94
23.39
37.23
32.39
61.52
56.74
59.45
87.92

1.54
23.99
38.05
23.34
37.25
32.77
61.67
57.43
61.66
73.27

1.54
24.04
38.33
23.24
37.49
32.55
61.75
57.19
62.97
67.99

3.91
24.05

2.72
48.67
42.06
53.64
48.55
50.56
33.27
33.23
8.37

3.22
49.97
53.15
66.60
50.16
66.90
51.54
51.53
16.86

3.43
50.68
62.04
68.17
65.04
78.72
54.48
54.48
33.07

3.60
51.28
62.83
68.78
76.28
79.38
70.34
70.33
51.28

3.72
51.58
63.37
69.18
76.92
79.92
82.23
82.30
54.29

3.79
52.01
63.57
69.96
77.02
79.88
84.11
84.10
70.73

3.81
51.78
64.28
70.40
77.34
80.80
85.05
85.03
82.92

3.83
51.96
64.59
70.43
78.27
81.29
85.60
85.59
84.59

3.83
64.59
70.46
78.73
81.35
86.61
86.59
85.56

64.58
78.71
81.28
87.22
87.17
86.00

78.75

13.35
11.12
7.49
0.54
0.43
0.26
0.07
0.26
0.02
0.09

26.03
20.92
14.27
1.07
0.82
0.51
0.13
0.52
0.05
0.19

40.11
37.38
25.34
2.14
1.62
1.03
0.26
1.03
0.10
0.38

46.52
37.73
32.57
4.17
3.11
2.04
0.52
2.02
0.18
0.75

61.36
46.92
31.91
8.33
5.91
3.91
1.06
3.98
0.38
1.47

71.51
52.85
41.13
13.23
9.94
7.57
1.75
7.84
0.63
2.84

60.11
49.97
43.66
25.46
12.75
11.21
3.96
15.22
1.31
5.52

62.62
51.73
44.40
47.36
19.01
20.49
7.19
28.90
2.63
10.56

64.01
52.71
45.70
49.94
19.58
35.59
11.24
39.35
4.50
19.66

64.19
53.00
46.48
63.00
22.23
37.91
14.82
48.53
6.70
33.61

64.71
53.25
46.89
62.44
23.87
46.05
15.02
54.89
8.91
42.98

64.86
53.46
47.10
63.02
24.09
52.52
16.07
63.86
10.12
45.51

3.65
2.71
9.04
2.39
2.97
2.18
0.60

7.32
5.32
17.87
4.59
5.81
4.29
1.19

8.62
10.07
34.83
8.96
11.57
8.52
2.37

11.78
13.88
48.31
16.13
22.43
16.72
4.61

12.19
23.14
50.03
29.04
42.98
31.74
9.03

13.42
24.09
63.49
41.23
59.82
44.33
16.13

14.13
28.85
78.25
42.44
63.38
46.29
29.33

14.51
32.18
79.18
45.37
65.26
59.59
41.50

14.57
32.43
83.13
48.62
74.03
70.69
42.93

14.67
32.83
85.21
49.93
79.92
77.62
48.96

14.68
32.87
86.36
49.59
80.39
81.80
54.35

14.70
33.12
86.46
50.37
82.07
82.02
54.93

1

2

4

8

16

Conv4 3
Conv5 1
Conv5 2
Conv5 3

0.01
0.00
0.09
0.01

0.02
0.01
0.19
0.02

0.03
0.02
0.37
0.03

0.04
0.04
0.74
0.07

ResNet v1.5
Conv1
Conv2 1
Conv2 2
Conv2 3
Conv2 4
Conv3 1
Conv3 2
Conv3 3
Conv4 1
Conv4 2
Conv4 3
Conv5 1
Conv5 2
Conv5 3

4.66
4.71
1.00
6.02
4.00
2.11
3.97
4.07
1.03
2.38
3.56
0.85
1.06
3.28

3.70
10.00
2.00
11.79
7.92
4.16
7.94
8.28
2.01
4.67
8.04
1.68
2.10
6.51

4.10
15.45
3.99
17.47
8.29
7.77
15.51
13.90
4.00
9.18
14.87
3.15
4.12
12.10

Inception v3
Conv1
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8
Conv9
Conv10

1.43
10.47
18.47
9.02
12.14
7.10
14.95
9.21
3.68
4.11

1.02
14.26
23.19
14.25
18.11
13.68
28.98
18.02
7.29
8.15

VGG16
Conv1 1
Conv1 2
Conv2 1
Conv2 2
Conv3 1
Conv3 2
Conv4 1
Conv4 2
Conv5 1

2.30
41.87
40.21
43.70
34.34
45.16
16.99
16.94
4.13

SSD v1.1
Conv1
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8
Conv9
Conv10
AlexNetConv1
MaxPool1
Conv2
MaxPool2
Conv3
Conv4
Conv5

37.49
32.72
61.78
57.56
62.78
64.01

87.14
87.28
87.02

5.2. CONVOLUTION

81

Table 5.5: Convolution performance (arithmetic throughput, TFlops/s) of one IPU, in half precision, over our reference basket of convolutional neural network architectures. Missing data
represents experiments that did not fit in device memory.

Layer name

Batch size
4
8

1

2

7.45
30.60
0.18
0.87
0.37
1.57
5.81
2.54
3.22
10.28
5.91
2.74
7.77
4.84

14.23
39.87
0.23
1.32
0.75
2.62
10.97
4.27
5.85
15.78
9.66
4.82
11.16
8.45

15.04
47.74
0.28
1.62
0.92
3.90
16.30
6.19
9.88
20.59
14.12
8.74
15.91
13.36

11.95
55.50
0.30
1.86
1.03
5.19
21.39
7.99
15.12
33.03
20.20
14.41
25.60
18.98

ResNet-50 v1.5
Conv1
7.45
30.58
Conv2 1
11.85
Conv2 2
Conv2 3
34.44
Conv2 4
27.48
16.23
Conv3 1
Conv3 2
29.82
Conv3 3
23.16
Conv4 1
10.92
26.53
Conv4 2
Conv4 3
20.65
10.30
Conv5 1
22.52
Conv5 2
Conv5 3
21.39

14.23
39.91
18.92
45.79
35.51
22.59
38.20
32.72
15.88
32.99
27.09
15.56
29.48
27.91

15.04
47.74
26.73
54.30
44.54
27.89
47.03
41.29
21.75
41.86
34.94
20.84
32.88
32.84

11.95
53.98
36.84
65.21
51.39
35.41
53.89
51.86
28.29
47.94
45.20
26.60
44.85
42.66

2.66
58.19
69.61
45.06
52.82
41.10
58.89
50.85
45.56

2.82
70.92
70.25
51.92
61.51
46.42
59.99

3.17
71.59
67.43
57.36
65.65
55.94
62.12

49.27

41.35

7.20
69.21
70.71
69.25
70.20
69.13

7.08

ResNeXt
Conv1
Conv2 1
Conv2 2
Conv2 3
Conv2 4
Conv3 1
Conv3 2
Conv3 3
Conv4 1
Conv4 2
Conv4 3
Conv5 1
Conv5 2
Conv5 3

Inceptionv3
Conv1
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8
Conv9
Conv10

2.90
44.61
55.98
37.05
43.30
32.79
51.80
44.93
36.64

VGG16
Conv1 1
Conv1 2
Conv2 1
Conv2 2
Conv3 1
Conv3 2
Conv4 1
Conv4 2
Conv5 1

6.33
75.04
68.90
71.79
64.05
65.29
57.55
57.56
41.40

5.36
74.89
72.60
70.87
69.31
72.10
58.53
58.54
46.76

SSD v1.1
Conv1
Conv2
Conv3
Conv4

40.53
32.90
14.78

47.34
40.36
22.21

16

32

64

7.75
55.93
0.32
2.58
1.14
6.90
36.13
10.22
28.21
45.69
31.57
28.05
37.14
33.41

17.82

62.07
0.32
2.10
1.10
6.23
27.70
9.33
22.02
41.61
25.63
20.22
31.31
27.61

49.81

7.75
55.82
52.39
73.43
58.55
54.87
58.13
62.39
44.41
52.78
60.92
41.43
52.71
57.72

5.70

7.08

62.32
46.03
70.79
56.99
43.79
56.63
59.07
36.09
52.66
56.45
34.44

65.66
61.90

69.03
69.07

47.72

55.93
72.00
43.21
60.86
55.88
45.49
55.60
50.36
57.94
51.27
50.35
55.14

63.55
35.80
61.71
49.05
43.58
45.57

66.61

57.01
35.26

47.58

0.49
0.51
1.83
1.87
1.72
37.18
50.12
37.87
45.81
51.85
47.44

17.82

70.41

56.39
56.39

48.38
30.26

49.85
63.77
50.70
58.10
49.13
43.23
42.81

0.49
0.71
0.46
2.17
10.78
3.91
33.75
45.26
35.93
35.76
45.06
41.51

128

56.07
55.88
49.92

46.93

53.35
54.83
55.19
46.76

56.07
56.07
42.77

59.29
66.37
57.06
62.66
45.58
58.85
51.51
54.69
55.72
49.28
53.44

82

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES
Layer name

Batch size
4
8

1

2

Conv5
Conv6
Conv7
Conv8
Conv9
Conv10

6.62
5.30
1.08
4.51
0.43
1.47

10.49
8.37
2.09
7.07
0.81
2.77

16.08
12.45
3.88
10.51
1.53
4.62

22.04
18.04
7.13
14.83
2.94
6.68

AlexNetConv1
MaxPool1
Conv2
MaxPool2
Conv3
Conv4
Conv5

8.40
20.22
44.03
24.40
28.05
28.62
12.20

9.75
28.43
57.30
28.69
37.17
34.72
16.15

11.14
37.38
62.83
36.60
45.68
42.61
20.35

11.15
43.72
64.34
44.66
52.25
51.43
26.10

16

32

64

28.92
26.01
11.23
23.64
5.51
11.68

35.19
31.83
16.21
30.16
8.81
17.56

43.25
38.45
22.95
37.25
13.24
23.35

20.04
51.73
61.69
46.32
53.70
52.91
41.77

17.07
47.83
60.52
42.26
51.04
50.10
44.11

51.12
49.21
33.43

128

5.3. PSEUDO-RANDOM NUMBER GENERATION

5.3

83

Pseudo-Random Number Generation

We compare IPU and GPU performance at bulk pseudo-random number generation (PRNG) in terms of aggregate throughput on a per-chip basis. We findthat, thanks to its dedicated, in-core PRNG hardware units, the IPU generates up to 944 billion random samples per second (Gsamples/s), whereas a
V100 GPU generates up to 192 Gsamples/s. While the IPU offers 4.9× moreaggregate throughput than a V100 GPU, it employs a PRNG algorithm thatdelivers a lower quality of randomness than the fastest algorithm we benchmarked on the GPU. The IPU’s performance advantage over the GPU doubles
in a per-board comparison. We are not qualified to judge which platform theperformance-quality trade-off favors. All details follow.
For simplicity, our benchmarks focus only on the generation of pseudorandom numbers extracted from a uniform distribution.
PRNG on the IPU. On the IPU, each tile includes PRNG acceleration
circuitry that implements a variant of the xoroshiro128+ algorithm by DavidBlackman and Sebastiano Vigna [15]. Details on the algorithm are available
upon request from Graphcore. Our benchmark invokes the poprand::uniformfunction from the Poplar SDK. For sufficiently large blocks, the aggregate IPU
throughput offered by the function reaches 944 Gsamples/s; see Figure 5.4.
PRNG on the GPU. On the GPUs, our benchmarks exercise PRNG func-

Throughput (Gsamples/s)

1000
IPU
V100
T4

800

600

400

200

0

IPU
xoroshiro128aox

GPU
Philox4GPU
XORWOWGPU
MRG32k3aGPU
MT19937

Platform and Algorithm
Figure 5.4: Bulk random number generation performance on IPU and GPUs: Peak generation
throughput in billions random samples per second, uniform distribution. Per-chip comparison.

84

CHAPTER 5. NOTABLE ARITHMETIC PRIMITIVES
1000

900

800

IPU
T4_MRG32K3A
T4_MT19937
T4_PHILOX4T4_XORWOW
V100_MRG32K3A
V100_MT19937
V100_PHILOX4
V100_XORWOW

700

Throughput (Gsamples/s)

600

500

400

300

200

100

0
1K

4K

16K

64K

256K

1M

4M

16M

Pseudo random sequence size

Figure 5.5: Bulk random generation throughput on IPUs and GPUs as a function of block size.
Per-chip comparison.

5.3. PSEUDO-RANDOM NUMBER GENERATION

85

tions offered with NVidia’s cuRand[16] library. cuRand supports multiple PRNG algorithms; we benchmark XORWOW, MRG32K3A, MTGP32,
MT19937, Philox4. The fastest algorithm at volume on the V100 GPU is
Philox4[17] by John Salmon. We run the experiment on each GPU (V100 and
T4) at the maximum clock frequency supported by each device.
A comparison on the quality of randomness provided by PRNG algorithms (e.g., TestU01[18] performance) is beyond the scope of this paper.
Output size. We study how bulk PRNG performance varies as a functionof requested output block size. Our results show that both on IPUs and GPUs,
the generation of larger output blocks achieves higher throughputs. On thetwo platforms, performance grows and saturates similarly, as a function of
output size (Figure 5.5)

List of Figures

1.1

2.1

2.2

2.3

3.1

4.1

4.2
4.3
4.4

4.5

Simplified block diagram of an IPU processor: the processor features 1,216
tiles (each containing one core and its local memory), the exchange (an onchip interconnect), IPU link interfaces that provide connectivity to other
IPU chips, and PCIe interfaces for host connectivity. . . . . . . . . . . . . .
Topology and interconnection of C2 boards and IPU processors in the system that we employed in all our experiments. All IPU numbers in thisfigure are Device IDs. Board placement in the chassis is depicted accurately(see Fwd and Aft arrows). . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Detailed tile layout within an IPU processor, including tile numbering. The
logical tile IDs depicted are those exposed to the programmer. Tiles topologically close to each other actually experience shorter communication latencies than tiles farther away (see Section 4.1.2). Source: direct communication with Graphcore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Topology of Multi-IPU virtual devices (depicted in red) with respect to thephysical IPUs (in black). All numbers are Device IDs. For example, IPU16
is a virtual device containing physical IPUs 5 and 7. IPU30 is a virtual
device containing all 16 physical IPUs in the server. . . . . . . . . . . . . .
Aggregate local memory bandwidth on the entire IPU as a function of block
size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Communication topologies we evaluate in point-to-point latency benchmarks. Data transfers between the source and the destination tiles are depicted with blue arrows. The experiments are designed to exercise both
on-chip (a) and off-chip (b,c,d) interconnects. . . . . . . . . . . . . . . . . .
Minimum latency between all tiles belonging to the same Column on an
IPU processor (we first presented columns in Figure 2.2). . . . . . . . . . .
Minimum latency between all pairs of tiles on an IPU processor. . . . . . .
Minimum latency between each pair of IPUs in our experimental 16-IPU
system, measured in zero-congestion conditions. IPUs are numbered according to their DNC IDs, as discussed in Section 2.2. . . . . . . . . . . . . .
The effect of physical proximity on tile-to-tile transfer bandwidth within
an IPU is negligible, especially with large messages. In our experiments,
the chosen pair of neighboring tiles is (0,1) and the chosen pair of far tiles
is (0,644), consistent with the tile enumeration of Figure 2.2. . . . . . . . . .

86

8

18

20

22

29

32
34
35

36

40

List of Figures
Peak monodirectional bandwidth between pairs of IPUs, in congestion-free
conditions. IPUs are numbered according to their DNC IDs, as discussed
in Section 2.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7 Topologies we benchmark in our broadcast and scatter experiments. Different topologies have different connectivity between source and destination tiles and perform differently. We depict data flows from the source tothe destination tiles with blue arrows. Reversing all blue arrows depicts
the data flows in the gather experiments. . . . . . . . . . . . . . . . . . . .
4.8 Broadcast latency scaling. Left: scaling within an IPU (on chip). Right:
scaling and across multiple IPUs (off chip). . . . . . . . . . . . . . . . . . .
4.9 Impact of block size on aggregate broadcast bandwidth. Each line represents a different experiment topology (a)-(j) as illustrated in Figure 4.7. . .
4.10 Impact of block size on broadcast bandwidth per tile. Each line represents
a different experiment topology (a)-(j) as illustrated in Figure 4.7. . . . . . .
4.11 Effect of operand size on reduction bandwidth in weak scaling conditions.
IPU identifiers are DNC IDs. . . . . . . . . . . . . . . . . . . . . . . . . . . .

87

4.6

5.1

5.2
5.3
5.4

5.5

Floating-point arithmetic throughput achieved by dense matrix multiplication on one IPU. For simplicity, we use square inputs (n × n) × (n × n);
the horizontal axis varies n. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Matrix multiplication performance on IPUs and GPUs. . . . . . . . . . . . .
Energy efficiency while running matrix multiplication on IPUs and GPUs.
Bulk random number generation performance on IPU and GPUs: Peakgeneration throughput in billions random samples per second, uniform
distribution. Per-chip comparison. . . . . . . . . . . . . . . . . . . . . . . .
Bulk random generation throughput on IPUs and GPUs as a function ofblock size. Per-chip comparison. . . . . . . . . . . . . . . . . . . . . . . . .

43

45
48
52
53
66

74
76
77

83
84

Bibliography[1]

Z. Jia, M. Maggioni, B. Staiger, and D. P. Scarpazza, “Dissecting theNVidia Volta GPU architecture via microbenchmarking,” 2018. [Online].
Available: https://arxiv.org/abs/1804.06826[2]

Z. Jia, M. Maggioni, J. Smith, and D. P. Scarpazza, “Dissecting theNVidia Turing T4 GPU architecture via microbenchmarking,” 2019.
[Online]. Available: https://arxiv.org/abs/1903.07486[3]

D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos,
R. Subramonian, and T. von Eicken, “Logp: Towards a realistic model ofparallel computation,” in Proceedings of the Fourth ACM SIGPLAN
Symposium on Principles and Practice of Parallel Programming, ser. PPOPP’93. New York, NY, USA: ACM, 1993, pp. 1–12. [Online]. Available:
http://doi.acm.org/10.1145/155332.155333[4]

A. Alexandrov, M. F. Ionescu, K. E. Schauser, and C. Scheiman, “Loggp:
Incorporating long messages into the logp model — one step closer
towards a realistic model for parallel computation,” Santa Barbara, CA,
USA, Tech. Rep., 1995.

[5]

J. Liu, B. Chandrasekaran, W. Yu, J. Wu, D. Buntinas, S. Kini, D. Panda,
and P. Wyckoff, “Microbenchmark performance comparison ofhigh-speed cluster interconnects,” IEEE Micro, vol. 24, pp. 42 – 51, 02
2004.

[6]

M. Kistler, M. Perrone, and F. Petrini, “Cell multiprocessor
communication network: Built for speed,” IEEE Micro, vol. 26, no. 3, pp.
10–23, May 2006. [Online]. Available:
http://dx.doi.org/10.1109/MM.2006.49[7]

L. G. Valiant, “A bridging model for parallel computation,”
Communications of the ACM, vol. 33, no. 8, pp. 103–111, Aug. 1990.
[Online]. Available: http://doi.acm.org/10.1145/79173.79181[8]

The Ohio State University’s Network-Based Computing Laboratory,“Osu micro benchmarks,” 2019. [Online]. Available:
http://mvapich.cse.ohio-state.edu/benchmarks/
89

90
[9]

BIBLIOGRAPHY
S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He, “Aggregated residualtransformations for deep neural networks,” 2016.

[10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” 2015.
[11] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
the inception architecture for computer vision,” 2015.
[12] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” 2014.
[13] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.
Berg, “Ssd: Single shot multibox detector,” Lecture Notes in Computer
Science, pp. 21–37, 2016. [Online]. Available:
http://dx.doi.org/10.1007/978-3-319-46448-0 2[14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” pp. 1097–1105, 2012.
[15] D. Blackman and S. Vigna, “Scrambled linear pseudorandom number
generators,” preprint article, 2019. [Online]. Available:
http://vigna.di.unimi.it/ftp/papers/ScrambledLinear.pdf[16] NVidia, “cuRand – the API reference guide for cuRand, the CUDA
random number generation library,” 2019. [Online]. Available:
https://docs.nvidia.com/cuda/curand/index.html[17] J. Salmon, M. Moraes, R. Dror, and D. Shaw, “Parallel random numbers:
As easy as 1, 2, 3,” 11 2011, p. 16. [Online]. Available:
https://thesalmons.org/john/random123/papers/random123sc11.pdf[18] P. L’Ecuyer and R. Simard, “TestU01: A c library for empirical testing ofrandom number generators,” ACM Transactions on Mathematical
Software, vol. 33, 2007.

The Authors
Zhe Jia is a senior R&D engineer with the High-Performance
Computing group at Citadel. Prior to this position, he was asenior R&D engineer with Alicloud, and a software engineering intern at Microsoft, Beijing. He received his B.S. degree inPhysics and his M.S. degree in Meteorology from Peking University, China. His interests include the performance optimizationof deep learning, numerical modeling, and atmospheric simulation workloads.
Blake Tillman is a Software Engineer with the HighPerformance Computing group at Citadel. He joined the firmin July 2019, after receiving a B.S. degree in Computer Scienceand a B.S. degree in Mathematics. Prior to joining Citadel, Blake
completed internships at Google and Apple.
Marco Maggioni is a senior R&D engineer with the HighPerformance Computing group at Citadel, Chicago. He received
his Ph.D. in Computer Science from the University of Illinois
at Chicago, where he focused on sparse linear algebra and convex optimization on GPUs. His research established records forthe fastest sparse matrix-vector multiplication GPU kernel in research and industry.
Daniele Paolo Scarpazza leads the High-Performance Computing group at Citadel, Chicago. Prior to this position, he was a Research Scientist with D. E. Shaw Research, a Research Staff Member with the IBM T. J. Watson Research Center, and a Post-Doc
with the Pacific Northwest National Laboratory. He received his
Ph.D. in Information Engineering from Politecnico di Milano,
Italy. He is the co-recipient of a Gordon Bell prize. He focuseson quantitative performance analysis and optimization of algorithms on parallel architectures.

91