45TH ADVANCED SUPERCOMPUTING ENVIRONMENT SEMINAR

Data-movement
accelerators (DMXs)

Richard (Rich) Vuduc — November 29, 2023
These slides + links to papers:

hpcgarage.org/ase45

Host
Mem

NIC

$
xPU

Network

45TH ADVANCED SUPERCOMPUTING ENVIRONMENT SEMINAR

Data-movement
accelerators (DMXs)

Richard (Rich) Vuduc — November 29, 2023
These slides + links to papers:

hpcgarage.org/ase45

Host
Mem

NIC

$
xPU

Network

45TH ADVANCED SUPERCOMPUTING ENVIRONMENT SEMINAR

Data-movement
accelerators (DMXs)

Richard (Rich) Vuduc — November 29, 2023
These slides + links to papers:

hpcgarage.org/ase45

Host
Mem
$
xPU

!NIC
mem
$
ypu

Network

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )“Smart” NICs are an old
idea…

N.J. Boden et al. “Myrinet: a gigabit-per-second local area network.” IEEE Micro, 15(1):29–36,
1995. doi:10.1109/40.342015
F. Petrini et al. “The Quadrics network: high-performance clustering technology.” IEEE Micro,
22(1):46–57, 2002. doi:10.1109/40.988689
R. Brightwell et al. “SeaStar interconnect: balanced bandwidth for scalable performance.” IEEE
Micro, 26(3):41–47, 2006. doi:10.1109/MM.2006.65

200 MHz

10

FIFO
queue
0

Link
multiplexer

72

Threadprocessor

SDRAM
interface64

Inputter

Data bus

Four-way
set-associative cache

FIFO
queue
1

Microcode
processor
Direct memory
access engine

64

10

100 MHz

32

28

Memorymanagement unit
and translation
look-aside buffer

Table
walk
engineClock andstatistics
registers

PCI interface

64

Figure 1. Elan functional units.

to-end protocols that detect faults and automatically retransmit packets.

nal functional structure of Elan, shown in Figure 1, centers around two primary processing
engines: the microcode processor and the66 MHzDATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )… born again in a newcontext.

Streaming data and latency-sensitive, in-transit processing are
the hallmarks of modern data center workloads.

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Today’s smartNIC landscape:a panoply of designs
💡NIC
mem
$
ypu

6

45TH ASE SEMINAR (2023) — TŌKYŌDATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Today’s smartNIC landscape:a panoply of designs
💡NIC
mem
$

7

45TH ASE SEMINAR (2023) — TŌKYŌ

ASICs,ypu
FPGAs,
multicore CPUs,
GPUs, …

age
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )BlueField-2 DPU - 2x 25Gb/s HHHL
form factor

Today’s smartNIC landscape:a panoply of designs
💡NIC
mem
$

BlueField-2 DPU - 2x 100Gb/s FHHL
form factor
845TH ASE SEMINAR (2023) — TŌKYŌ

ASICs,ypu
FPGAs,
multicore CPUs,
GPUs, …

age
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )BlueField-2 DPU - 2x 25Gb/s HHHL
form factor

Today’s smartNIC landscape:a panoply of designs
💡NIC
mem
$
ypu
BlueField-2 DPU - 2x 100Gb/s FHHL
form factor“DPU”(data processing unit)
945TH ASE SEMINAR (2023) — TŌKYŌ

age
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )BlueField-2 DPU - 2x 25Gb/s HHHL
form factor

Today’s smartNIC landscape:a panoply of designs
💡NIC
mem
$
ypu
BlueField-2 DPU - 2x 100Gb/s FHHL
form factor
1045TH ASE SEMINAR (2023) — TŌKYŌ

“DPU”
BF-2: 8-core Arm v8 A72 @ 2.6 GHz,
DDR4 4800 MT/s, HDR100 @ 100 Gb/s
BF-3: 16-core Arm v8 A78 @ 2.25 GHz,
DDR5 5600 MT/s, NDR200 @ 200 Gb/s

BF-4
BlueField-4
NVIDIA DPU ROADMAP
64B64
Transistors
B
Exponential Growth in Data
CenterInfrastructure
Processing
Exponential Growth in Data Center Infrastructure Processing 160 SPECint*
Top/s
https://hc33.hotchips.org/assets/program/conference/day1/HC2021.NVIDIA.IdanBurstein.v08.norecording.pdf
400
1000
TOPS
100X
BlueField-4
800 Gbps
64B Transistors
Gb/s

NVIDIA DPU ROADMAPBlueField-4
64B Transistors
160 SPECint*
1000 TOPS
800 Gbps

160 SPECint*
1000 TOPS
800 Gbps

100X

100X

BF-3
BlueField-3
B
22
22B
Transistors
BlueField-3
42 SPECint*
Top/s
1.5
22B
Transistors
1.5
TOPS
42 SPECint*
Gb/s
1.5400TOPSGbps
400

BlueField-3
22B Transistors
42 SPECint*
1.5 TOPS
400 Gbps

10X

10X
10X

BlueField-2
7B Transistors
9 SPECint*
0.7 TOPS
200 Gbps

1X

400 Gbps

BF-2
2024
2022
BlueField-2
BlueField-2
B7Btransistors
Transistors
7
7B Transistors
9 SPECint*
1X * SPECint2k17-rate
0.7 TOPS
9 SPECint*
0.7 Top/s
200 Gbps
0.7 TOPS
Gb/s
200
DOCA — ONE DEVELOPMENT ARCHITECTURE
200 Gbps

DOCA — ONE DEVELOPMENT ARCHITECTURE

2020

5

1X

2020

2020

2022

DOCA — ONE DEVELOPMENT ARCHITECTURE

2022* SPECint2k17-rate

2024

2024

800

Q: Are smartNICs for data
centers relevant to HPC?

1245TH ASE SEMINAR (2023) — TŌKYŌ

Claim: Communication isan inevitable bottleneck

13

45TH ASE SEMINAR (2023) — TŌKYŌ

Recall: “The” dominant paradigm of CS:<latexit sha1_base64="wlvbpO/iCFawUxEqvXghJqsRc90=">AAAGhnicdVTdbts2FFbb1cu8v7a73A07o0AKqLElx3aSq2zulgxrZ89t2gKWl9H0kSyEogSSiusSeoA9zW63R9nb7FC2MStxCUggeb7znV+eacZjpVutf+/cvffJ/dqne5/VP//iy6++fvDw0RuV5pLBBUt5Kt9NqQIeC7jQsebwLpNAkymHt9OrvpW/vQap4lS81ssMJgmNRBzGjGq8unzQCKZxNDC//u4XJOCpiGQczTWVMl2QtahAVOugVS5ye+OtNw1nvYaXD+//EsxSlicgNONUqbHXyvTEUKljxqGoB7mCjLIrGoGhiVLLZFqQJwnVc3VTZi93yca5Do8mJhZZrkEwhKAszDnRKbGBklksgWm+xA1lMkbLhM2ppExjOur1J+QlvQJyDvwaUEaJnqMOhDTnumJHMcphNjHzEom+SxCwYGmSUDELQprEfLnWM4EK19uCVEheexMTpqL01Nr+UVCsEJHAsQ7XQFT8IRZRJXaU4S2U8O9znWIGMIRYKJC2dCQNCbzXkhKFGlDN2/vyrlR9DiG2Bjkf9skZRh8BKZuGKDaHBMizZ2Q8xAC1BtIkAwZUkJ+sRycnpD94MbgY8dT2z2R/rnWmTprNxWJxYClyuZLgIWlmK4qmd9hrN0uWS8vytOpVaRkzOCt9Kk8GHTubYreJwozOfiiM1/LdXsftHO/C8RzWsG7b9TrHrtfr7cBJmG3Yjn338MjtHu5ApZKKaMPn+0jY6rhdfwdyCZyniw2y7bl+q+t22zuQkYTlxnS3iy4e2W8HcEbl1Rb4uIPWe67n23D+by8TzDMWlVq2Fcb+xJhgRdLwioZfBJtKV5QYl2Wqxth1FYatLKL+bS2buN1KZUpLHeypoYIc33c6A+xH0n8xetXGFtZLDpV6I6GyoDbceDUmiECrwgQcwtWsKWn7KMXGtk+d2JGQclX10M6jlX8gVC7BIk1g//hGzaAIHpeM+w0vKMfY04/kZwSUW/PbLDIhPwePR1sazwFnl4SXKB5kIKnGrAdCfCiM/dU/AhjKdFqYobSAbZvXVL7Kp2UJt+3anF6aVRAyMYGOxZJgYXFtPMHx690ctrc3b/wDr3vQ+e2wcfrHehDvOd863zn7juf0nFPn3Bk6Fw5z/nT+cv52/qnt1Q5qnVpvBb17Z63zjVNZtdP/AKYCPQM=</latexit>

O N

2! O(N )

fl

Reduces energy: fewer ( )ops, less storageRecall: “The” dominant paradigm of CS:<latexit sha1_base64="wlvbpO/iCFawUxEqvXghJqsRc90=">AAAGhnicdVTdbts2FFbb1cu8v7a73A07o0AKqLElx3aSq2zulgxrZ89t2gKWl9H0kSyEogSSiusSeoA9zW63R9nb7FC2MStxCUggeb7znV+eacZjpVutf+/cvffJ/dqne5/VP//iy6++fvDw0RuV5pLBBUt5Kt9NqQIeC7jQsebwLpNAkymHt9OrvpW/vQap4lS81ssMJgmNRBzGjGq8unzQCKZxNDC//u4XJOCpiGQczTWVMl2QtahAVOugVS5ye+OtNw1nvYaXD+//EsxSlicgNONUqbHXyvTEUKljxqGoB7mCjLIrGoGhiVLLZFqQJwnVc3VTZi93yca5Do8mJhZZrkEwhKAszDnRKbGBklksgWm+xA1lMkbLhM2ppExjOur1J+QlvQJyDvwaUEaJnqMOhDTnumJHMcphNjHzEom+SxCwYGmSUDELQprEfLnWM4EK19uCVEheexMTpqL01Nr+UVCsEJHAsQ7XQFT8IRZRJXaU4S2U8O9znWIGMIRYKJC2dCQNCbzXkhKFGlDN2/vyrlR9DiG2Bjkf9skZRh8BKZuGKDaHBMizZ2Q8xAC1BtIkAwZUkJ+sRycnpD94MbgY8dT2z2R/rnWmTprNxWJxYClyuZLgIWlmK4qmd9hrN0uWS8vytOpVaRkzOCt9Kk8GHTubYreJwozOfiiM1/LdXsftHO/C8RzWsG7b9TrHrtfr7cBJmG3Yjn338MjtHu5ApZKKaMPn+0jY6rhdfwdyCZyniw2y7bl+q+t22zuQkYTlxnS3iy4e2W8HcEbl1Rb4uIPWe67n23D+by8TzDMWlVq2Fcb+xJhgRdLwioZfBJtKV5QYl2Wqxth1FYatLKL+bS2buN1KZUpLHeypoYIc33c6A+xH0n8xetXGFtZLDpV6I6GyoDbceDUmiECrwgQcwtWsKWn7KMXGtk+d2JGQclX10M6jlX8gVC7BIk1g//hGzaAIHpeM+w0vKMfY04/kZwSUW/PbLDIhPwePR1sazwFnl4SXKB5kIKnGrAdCfCiM/dU/AhjKdFqYobSAbZvXVL7Kp2UJt+3anF6aVRAyMYGOxZJgYXFtPMHx690ctrc3b/wDr3vQ+e2wcfrHehDvOd863zn7juf0nFPn3Bk6Fw5z/nT+cv52/qnt1Q5qnVpvBb17Z63zjVNZtdP/AKYCPQM=</latexit>

O N

2! O(N )

% time communicating increases

fi

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )An Iron Law of Parallel andDistributed Computation
A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

16

45TH ASE SEMINAR (2023) — TŌKYŌ

Two costs: Tnetwork + Tmemory

fi

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )An Iron Law of Parallel andDistributed Computation
A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

17

45TH ASE SEMINAR (2023) — TŌKYŌ

Two costs: Tnetwork + Tmemory

fi

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )An Iron Law of Parallel andDistributed Computation
A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

MemoryMemoryMemory

$

$

$

xPU

xPU

xPU

“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs
between nodes across the network.

18

45TH ASE SEMINAR (2023) — TŌKYŌ

Two costs: Tnetwork + TmemoryfiCompute time

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )An Iron Law of Parallel andDistributed Computation
A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.
“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

19

45TH ASE SEMINAR (2023) — TŌKYŌ

Network time<latexit sha1_base64="eI+6GmJxQ5XUXgCUxV9muqTZut8=">AAAHSXicdVVfb9s2EJfbNe28P023x72wMzKkgBpbcmwneeqWbsmwFPbSpC1gGQEln2QhFKWQVByH0CfcJ9gH2AfY27CnHSV7tROXgMUT73e/O90dz37GYqlarT9rDx5+9mjj8ZPP6198+dXXTzefffNOprkI4DxIWSo++FQCizmcq1gx+JAJoInP4L1/eWj0769ByDjlZ2qWwSihEY/DOKAKjy42//I4TIM0SSgfa+9M5n4xdEbau6bibe7rs0I3nKKor6JQyooKrdPsnjqBZKE14l01B7VQG7Go1z0fophruOJUCDor6mfkB+JFcGW2hN4Qj0GoiKdJ5dsmpROzIQPxRBxNcCuIx1OeJz6Iugfo63/Ci81Ga6dVLnJfcOZCw5qvwcWzRyfeOA3yBLgKGJVy6LQyNdJUqDhggDFvkTf0EsgxsGvAM0rUBMgYQpozVfdyCRkNLmkEQxlQBuORnpRITIaAj+nwQprEbDa3054M52JBVkjOsCZhyhXwwOTro0bTRMpZ4hdkK6FqIu/qzOE63TBX4d5IxzzLS1KyhbowZ0SlxPQJGccCAsVmKNBAxPiJJJhQQQOF3WQ+/2dOscmIAIatdA1Exrcxj1b8ow5Pq2z9mKsUo0CamEsQpvtIGhK4UYISiRawGvtNeVaavoYQu5scDw7JEUYQASn7nshgAgmQly/JcIA5VgpIk/QDoJz8YiI6OCCH/ZP++SlLzRUYbU+UyuRBszmdTncMRS4qDb4kzayiaDq7vXazZLkwLC9Woyo9YxHHZUzlm8bAjnyRTnmhT49+KrTTcu1ex+7sr8OxHOawbtt2Ovu20+utwQkYL9j2XXt3z+7urkGlgvJowee6SNjq2F13DXIGjKXTBbLt2G6ra3fba5CRgNnCdbeLIe6Z3xrgmIrLJfB+B733bMft3bnwkyyISivTCkN3pLVXkeBkabiFt6j0ilHARJmqchgtMyxl8f5kQiuTuPVGZUpLG+ypgYQcL3g6BuxHcnhy+raNLaxmDFbqjYTSgNpw5+JqHE9K4hwzkwmHDObW0B6iFhvbXDdirmXK5GqEfhz1q/iAy1yAQWrPPHFM6H7hPS8ZtxtONdVefCI/p0CZcb/MIhLyq/f8dMniNeDwEvAG1f0MBFWYdY/z20KbR/0TgIFI/UIPhAEs+6z+E8oSLvs1Ob3Q1UfIUHsq5jOChcW1iATnr3N32t4X3rk7Tnen87vbeDWYT+In1nfW99a25Vg965V1bA2scyuo/Va7qt3W9MYfG39v/LPxbwV9UJvbfGutrMcP/wOrFImi</latexit>T

max {Top , Tmem , Tnet }Memory time

Memory
$

xPU

fi

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation
A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.
“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

20

45TH ASE SEMINAR (2023) — TŌKYŌ

T

network penalty

Tmem Tnet
Top max 1,,
Top Topmemory penalty

Memory
$

xPU

fi

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation

T

Tmem Tnet
Top max 1,,
Top TopA modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
As a program executes on this system, it
incurs two types of communication cost.
“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

21

45TH ASE SEMINAR (2023) — TŌKYŌ

Memory$xPU<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit>

R0

ops
time

fi

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation

Tmem Tnet
Top max 1,,
Top TopT

A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

As a program executes on this system, it
incurs two types of communication cost.
“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

22

45TH ASE SEMINAR (2023) — TŌKYŌ

words
time

B0

Memory$xPU<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit>

R0

ops
time

fi

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation

Tmem Tnet
Top max 1,,
Top TopT

A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.
<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

As a program executes on this system, it
incurs two types of communication cost.

words
time

B0

Memory$xPU“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

23

45TH ASE SEMINAR (2023) — TŌKYŌ

<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit>

R0<latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

words
time

ops
time

fi

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation

Tmem Tnet
Top max 1,,
Top TopT

A modern cluster or supercomputer is, torst order, a collection of processing nodes.
Each node has a processor (“xPU”) and a
two-level memory hierarchy. Nodes areconnected by a network.

Memory<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

As a program executes on this system, it
incurs two types of communication cost.
“Vertical” communication occurs in thememory system between, say, RAM andcache.
“Horizontal” communication occurs between
nodes across the network.

24

45TH ASE SEMINAR (2023) — TŌKYŌ

words
time

B0

$=Z
xPU
<latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit>

R0
words
time

ops
time

⇢

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )<latexit sha1_base64="xG8/hfPnnHLbBRYe4nITKqXBDm8=">AAAHaHicdVVfb9s2EJfbtem8P022h2HYC1sjQwu4sSXHdpKnbumWDEhhL03aAqYRUPJJ1kJRKknFcQl9h71u32xfYZ9iR9le7MQlYOl097vfHY/Hs5/xWOlm85/KvfufPXi48ejz6hdffvX1482tb96qNJcBnAcpT+V7nyngsYBzHWsO7zMJLPE5vPMvD6393RVIFafiTE8zGCYsEnEYB0yj6mKrcp8KmARpkjAxMvRM5X4xcIeGXjH5JvfNWWFqblFUV1EoZcUMbdLsjjmBZGG14m2zAL0wW7GoVqkPUSwMfBBMSjYtqmfkR0Ij+GBfZTRCE3ZNKIdQU0PcOqGhZMFNrDKjG+08RKklVMbRWFMURCryxAdZpYCZ/B/uYrPW3GmWi9wV3LlQc+arf7H14ISO0iBPQOiAM6UGbjPTQ8OkjgMOuKNt8ppdAjkGfgWoY0SPgYwgZDnXVZoryFhwySIYqIBxGA3NuERiqSTcFIuGLIn5dO5nqArnYkFWSM7wxMJUaBCBreaNxbBEqWniF2Q7YXqsbtuscp1tkOtwb2hikeUlKdlGW5hzolNi24iMYgmB5lMUWCBj3CIJxgyLr7HZ7PZ/EQx7kEjg2GlXQFT8MRbRSny0oXZWrZ9ynWIWSBMLBdI2J0lDAtdaMqLQA1Zzvy51pesrCLH5yXH/kBxhBhGQ8loQFYwhAfLiBRn0scZaA2mQXgBMkF9tRgcH5LB30js/5am9IcNnY60zddBoTCaTHUuRy5kFP5JGNqNouLvdVqNkubAsz1ezKiPjIY7KnMovg4kd+TKdiMKcHv1cGLfp1bvtent/HY7nMId1WnW3vV93u901OAmjBdu+V9/dq3d216BSyUS04PM8JGy26x1vDXIKnKeTBbLl1r1mp95prUFGEqaL0J0Oprhnf2uAIyYvl8D7bYzerbte99Y4GGdBVHrZVhh4Q2PojATnTs0r6OKkV5wCLstSlaNqmWGpinfnFnrZwq13Kkta+mBP9RXkeMHTEWA/ksOT0zctbGE95bBy3kioLKgFty6uweGlFY4gO7BwyGBtLe0hWrGx7XUj9lqmXK1m6MdRb5YfCJVLsEhD7RPHhOkV9EnJ+Kzmzoba80/U5xQYt+GXWWRCfqNPTpc8XgEOLwmv0dzLQDKNVadCfCyMfVQ/AejL1C9MX1rAcszZP0Z5hMtxbU0vzGwTKjRUx2JK8GBxLTLB+evenrZ3hbfejtvZaf/u1V7255P4kfOD89R55rhO13npHDt959wJKn9U/qz8Vfn74b8bmxvfbXw/g96rzH2+dVbWxtP/AFefk3s=</latexit>

An Iron Law of Parallel andDistributed Computation

Tmem Tnet
Top max 1,,
Top TopT

memory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

Tmem
Top

⇡

memory penalty

R01·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

MemoryZ

xPU⇡

R0 h1 (P )·Bnet h2 (n)network penalty
25<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit>

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

network penalty<latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

ops
time

words
time

45TH ASE SEMINAR (2023) — TŌKYŌ

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Lower is better

An Iron Law of Parallel andDistributed Computation

Tmem
Top

⇡

“smaller” processors)

Processor-memory
op:bytememory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

(

R0
1
·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

MemoryZ

xPU

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

⇡

R0 h1 (P )·Bnet h2 (n)network penalty
26

45TH ASE SEMINAR (2023) — TŌKYŌ

Processor-network
op:byte<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit><latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

ops
time

words
time

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Lower is better

An Iron Law of Parallel andDistributed Computation(“bigger” processors)

memory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

Tmem
Top

⇡

R0
1
·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

MemoryZ

xPU

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

⇡

R0 h1 (P )·Bnet h2 (n)network penalty
27

Increasing functions<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit><latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

ops
time

words
time

45TH ASE SEMINAR (2023) — TŌKYŌ

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Lower is better

An Iron Law of Parallel andDistributed Computation(“bigger” processors)

memory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

Tmem
Top

⇡

R0
1
·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

MemoryZ

xPU

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

⇡

R0 h1 (P )·Bnet h2 (n)network penalty
28

Network overhead<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit><latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

ops
time

words
time

45TH ASE SEMINAR (2023) — TŌKYŌ

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”

Memory and network
communication trade off!

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )An Iron Law of Parallel andDistributed Computation(under strong scaling)

memory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

Tmem
Top

⇡

R0
1
·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

MemoryZ

xPU

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

⇡

R0 h1 (P )·Bnet h2 (n)network penalty
29

Network overhead<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit><latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Bnet

ops
time

words
time

45TH ASE SEMINAR (2023) — TŌKYŌ

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Higher is better

An Iron Law of Parallel andDistributed Computation(

weak scaling)

memory penalty<latexit sha1_base64="1z1cr1QFwRLlffJ9sRLGnkprntQ=">AAAHtHicdVRtb9s2EJa7de6ylzbbx31hZ2RIBje25NhO8qlruiUDUthzk7aYZQSUfJKFUJRGUnFcQn9r/2XAvq6/o0fJXmzHJWDjxOe5547H43kpi6RqNv+pPPjs84dfVB99ufXV1998+/jJ9ndvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN561ycGf3sDQkYJv1CzFEYxDXkURD5VuHW1Xem5HKZ+EseUj7V7ITMvH9oj7d5Q8Trz9EWua3aeb62y0Erzkq2T9B4cQ7xAjbkOc1AL2Jhr8OB9/n/0Qa6b6/iLJfzFJryUv2OUMbZcD8KIa/iLUyHo7Gd0CwT179ItDkV+Ii5NU5HcGqskFBlhWOL640TNd+1ch7t/7qEKYNQ71asnteZ+s1jkvmHPjZo1X/2r7Yfn7jjxsxi48hmVcmg3UzXSVKjIZ4CZ75BX9BrIGbAbwD1K1ATIGAKaMbXlZhJS6l/TEIbSpwzGIz0pmJiagLvKuAGNIzab+2lXBnMzJysiF3j7QcIVcN9U7Q7RNJZyFns52Ympmsh1zGxuwoaZCg5HOuJpVoiSHcSCjBGVENOSZBwJ8BWboUF9EeERiT+hWGSFjWuO/yun2M9EAMOuvQEio/cRD1fiI4a7ZbV+yVSCWaBMxCUI0+gkCQjcKkGJRA9Yzf222CtcX0KAD4mc9U/IKWYQAimeGJH+BGIgz56RYR9rrBSQBun5QDn5zWR0fExOeue9ywFLzGsb7U6USuVxozGdTveNRCZKBD/iRlpKNOyDbqtRqFwZlb3VrIrIeInjIqfiS2Nip55IpjzXg1Psbbvp1LvtevtoE49lMKd1WnW7fVS3u90NPAHjhdqRUz84rHcONrASQXm40HMcFGy26x1nA3MGjCXTBbNl151mp95pbWCGAmaL0J0OpnhofhuIYyqul8hHbYzerdtOd+3tT1I/LLxMKwydkdZuKYIzrObk7uKmV5x8JopSFWNvWWGpivdnIHqZwm12Kkpa+GBP9SVk+MCTMWA/kpPzwesWtrCaMVi5bxSUhtSCtYer3RCUxBHEIFA4ZbC2RvYEUWxs89yIeZYJk6sZelHYK/MDLjMBhqld849jQvdy92mhuFuzXRGFE7X3ifoMgDITfllFxOR39+lgyeMl4PAS8ArhXgqCKqy6yzmOTvO39QlCXyRervvCEJZjltO7uMLluKamV7o8hAy0qyI+I3ixuBaZ4Py116ftfeONs2939tt/OLXn/fkkfmT9YP1o7Vq21bWeW2dW37q0/MrflX8r/1U+VDtVt+pXoaQ+qMx9vrdWVpV/BIGbtSs=</latexit>

Tmem
Top

⇡

R0
1
·
B0 g(Z)<latexit sha1_base64="6c2hLRPaGYexb313AyAgRK3IBso=">AAAHNnicdVVfc9pGEBdJQ1P6J0762JdLGWecDjFIGLD9lIa0dmecgTp2khlEmZNYCY1PJ/XuZEJu9L36OfrWl751+tqP0D0BMWByMzB7t7/97d7u3spLWSRVo/FX6c7dz+6VP7//ReXLr77+5sHOw0dvZJIJHy79hCXinUclsIjDpYoUg3epABp7DN56V12jf3sNQkYJv1CzFIYxDXkURD5VeDTa+dPlMPWTOKZ8rN0LmXn5wB5q95qK15mnL3JdtfO8so5CKc3naJ2kt9QxxEutETfVHNRSbcS8UnE9CCOu4XdOhaCzH/LKE+IGgvrajamayEBPEzGWef5xr6IY8pw8IS9GjYoLyHtjPNqpNvYbxSK3BXshVK3F6o8e3jtzx4mfxcCVz6iUA7uRqqGmQkU+Awxwl7yiV0BOgV0DnlGiJkDGENCMqYqbSUipf0VDGEifMhgP9aRA4s0F3NzdDWgcsdnCTrsyWIg5WSO5wAIECVfAfZOcG42msZSz2MvJbpGHTZ053KYbZCo4HOqIp1lBSnZRF2SMqISYpiDjSICv2AwF6osIr0j8CcUCKGwdc/2fOMWOIgIY9s01EBl9iHi45h91eDrP1o+ZSjAKpIm4BGFajSQBgfdKUCLRAtZjf1+cFaYvIcBWJqf9LjnBCEIgRZMT6U8gBvLsGRn0McdKAamTng+Uk59NRMfHpNs7612es8T0+3BvolQqj+v16XS6bygyMdfgJq6nc4q6fdBp1guWkWF5uh5V4RmLOC5iKnYaAzvxRDLluT4/eZFru+HUOq1a62gbjmWwgLWbNbt1VLM7nS04AeMl25FTOzistQ+2oBJBebjkcxwkbLRqbWcLcgaMJdMlsmnXnEa71m5uQYYCZkvX7TaGeGh+W4BjKq5WwEct9N6p2U5n43VPUj8srEwrDJyh1u6cBMdI1cndZaXXjHwmilQVk2eVYSWLt8cQWpnEbTcqUlrYYE/1JWT4wJMxYD+S7tn56ya2sJoxWKs3EkoDasLGw9VuCEri9GEQKJwymFtD20UtNrZ5bsQ8y4TJ9Qi9KOzN4wMuMwEGOR9hOCZ0L3cfF4x7VdsVUThRTz+Rn3OgzLhfZREx+cV9fL5i8RJweAl4hepeCoIqzLrL+Ydcm7/KJwB9kXi57gsDWPU5/wAUJVz1a3I60h/nsKsiPiNYWFzLSHD+2pvT9rbwxtm32/utX53q8/5iEt+3vrO+t/Ys2+pYz61Tq29dWn7psPRbKSxNyn+U/y7/U/53Dr1TWth8a62t8n//A+E/hT0=</latexit>

words
time

B0

Memory$xPU

R0<latexit sha1_base64="SiFeAa/j9TOYkwyXVreYkWi0y40=">AAAHvXicdVRtc9pGEBZpU1L3JXH7sV8uZdyxO8QgYcD2l6Yhrd0ZZ6DETjyDGOYkVqDx6aTenYzJjX5bf0d/QL+2f6F7EtSAiWYQq3uefXZvb2+9hIVS1et/lR598unjz8pPPt/54suvvn76bPebdzJOhQ9Xfsxice1RCSzkcKVCxeA6EUAjj8F776Zj8Pe3IGQY80s1T2AY0QkPg9CnCpdGu6Vrl8PMj6OI8rF2L2XqZQN7qN1bKt6mnr7MdMXOsp11FlpJVrB1nDyAI4iWqDE3YQ5qCRtzA+5/yP6P3s90fRN/tYK/2oYX8veMIsaO68Ek5Br+4FQIOv8R3QJB/ft88k2RH4hLk0TEd8YqCHlGuSxx/XGsFuvTkb3fO8jw39nnBygHGP5efvSsUj+s5w95aNgLo2Itnt5o9/GFO479NAKufEalHNj1RA01FSr0GeAW9sgbegPkHNgt4BolagpkDAFNmdpxUwkJ9W/oBAbSpwzGQz3NmZiagPsSuQGNQjZf+GlXBgszI2sil9gGQcwVcN+U7x7RNJJyHnkZ2YuomspNzCxuwwapCo6HOuRJmouSPcSClBEVE9ObZBwK8BWbo0F9EeIWiT+lWGuFHWy2/wun2NhEAMP2vQUiww8hn6zFRwxXi2r9nKoYs0CZkEsQpuNJHBC4U4ISiR6wnvtdvpa7voYAbxQ573XIGWYwAZLfNSL9KURAXrwggx7WWCkgNdL1gXLyq8no9JR0uhfdqz6LzbUb7k+VSuRprTabzQ6NRCoKBD+iWlJI1OyjdqOWq4yMysF6VnlkPMRxnlP+pTGxM0/EM57p/hk2uV13qu1mtXmyjcdSWNBajardPKna7fYWnoDxUu3EqR4dV1tHW1ixoHyy1HMcFKw3qy1nC3MOjMWzJbNhV516q9pqbGFOBMyXoVstTPHY/LYQx1TcrJBPmhi9XbWd9sYQmCb+JPcyrTBwhlq7hQgOs4qTucuTXnPymchLlc+/VYWVKj4chuhlCrfdKS9p7oM91ZOQ4gWPx4D9SDoX/bcNbGE1Z7B23igoDakBGxdXuxNQEmcRg0DhlMHaGtkOotjY5roRcy1jJtcz9MJJt8gPuEwFGKZ2zRvHhO5m7vNccb9iuyKcTNXBR+rTB8pM+FUVEZHf3Of9FY/XgMNLwBuEuwkIqrDqLuc4Q81r5yOEnoi9TPeEIazGLMZ4foSrcU1NR7rYhAy0q0I+J3iw+Cwzwflrb07bh8Y759BuHTZ/dyove4tJ/MT6zvre2rdsq229tM6tnnVl+aU/S3+X/in9W/6pDGVW5gX1UWnh86219pRn/wH9prho</latexit>

Tnet
Top

⇡

R0 h1 (P )·Bnet h2 (n)network penalty
30<latexit sha1_base64="5a1NZdmfdYjh3/fLKzCDiodJ/Ek=">AAAHNHicdVTfbxpHED7Shqb0R5z2sS+bIkdORQx3GLD9lJa0diVHUGInkTiE9o654+S9vevunglZ3b/Vv6OvlfpW9bV/Q2cPiAGTlUCzO998M/ft7Hgpi6RqNP4q3fvk0/vlzx58Xvniy6++frj36JvXMsmED1d+whLx1qMSWMThSkWKwdtUAI09Bm+8667xv7kBIaOEX6p5CqOYhjwKIp8qPBrv/elymPlJHFM+0e6lzLx8aI+0e0PFq8zTl7mu2nle2UShleYLtE7SO+4Y4pXXmNtuDmrlNmZeqbgehBHX8DunQtD5D3nlCRmMG+QJcQNBfe3GVE1lgLlknn/YqSgGQw7Iehs63qs2DhvFIncNe2lUreXqjx/dv3AniZ/FwJXPqJRDu5GqkaZCRT4DLG+fvKTXQM6B3QCeUaKmQCYQ0IypiptJSKl/TUMYSp8ymIz0tEBiaQJuv9wNaByx+TJOuzJYmjnZILlE+YOEK+C+kebWo2ks5Tz2crJfKLDtM4e7fMNMBccjHfE0K0jJPvqCjBGVENMSZBIJ8BWbo0F9EeEnEn9KUXiFjWM+/2dOsZ+IAIZdcwNERu8jHm7kRx+eLtT6MVMJVoE0EZcgTKORJCDwTglKJEbAZu3virMi9AUE2MjkvN8lZ1hBCKRocSL9KcRAnj0jwz5qrBSQOun5QDn5xVR0ekq6vYve1YAlpttHB1OlUnlar89ms0NDkYmFBzdxPV1Q1O2jTrNesIwNy9PNqorMeImToqZip7GwM08kM57rwdlPubYbTq3TqrVOduFYBktYu1mzWyc1u9PZgRMwWbGdOLWj41r7aAcqEZSHKz7HQcJGq9Z2diDnwFgyWyGbds1ptGvt5g5kKGC+St1uY4nH5rcDOKHieg180sLsnZrtdLbe9jT1wyLKtMLQGWntLkhwiFSd3F3d9EaQz0QhVTF31hnWVLw7hDDKCLc7qJC0iMGe6kvI8IEnE8B+JN2LwasmtrCaM9i4bySUBtSErYer3RCUxLnDIFA4ZVBbQ9tFLza2eW7EPMuEyc0KvSjsLeoDLjMBBrkYXjgmdC93HxeMB1XbFVE4VU8/os8AKDPp11lETH51Hw/WIl4ADi8BL9HdS0FQhaq7nL/PtfmrfATQF4mX674wgPWci/FfXOF6XqPpWH+YwK6K+JzgxeJaVYLz196etneN186h3T5s/eZUn/eXk/iB9Z31vXVg2VbHem6dW33ryvJL7ZJbglJQ/qP8d/mf8r8L6L3SMuZba2OV//sf63mEXA==</latexit><latexit sha1_base64="st34fjHCBY7gg+oxTXQfPjpTVTM=">AAAHQnicdVTfc9pGEBZJQ1P6I0772JdLGXecDjFIGLD9lBa3dmecgTp2khnEMCexEopPJ/XuZEJu9N/1n+hL/4C+dfrah+4JiAGTm4HZu/3229Xed+ulLJKq0fizdO/+Jw/Knz78rPL5F19+9Wjn8devZJIJH678hCXijUclsIjDlYoUgzepABp7DF57113jf30DQkYJv1SzFIYxDXkURD5VeDTa+cvlMPWTOKZ8rN1LmXn5wB5q94aKl5mnL3NdtfO8so5CK83naJ2kd9wxxEuvMTfdHNTSbcy8UnE9CCOu4XdOhaCzH/IK+Z4sS/gpL2B4EAjqazemaiIDPU3EWOb5h72KYjCpAHPcEo12qo39RrHIXcNeGFVrsfqjxw/O3XHiZzFw5TMq5cBupGqoqVCRzwCL3SUv6DWQM2A3gGeUqAmQMQQ0Y6riZhJS6l/TEAbSpwzGQz0pkFiagNs+uAGNIzZbxGlXBgszJ2skl3gZQcIVcN806tajaSzlLPZyslt0YNNnDrf5BpkKDoc64mlWkJJd9AUZIyohRiBkHAnwFZuhQX0R4ScSf0Kx9QplZD7/Z05RXUQAQw3dAJHR+4iHa/nRh6fzbv2YqQSrQJqISxBGdiQJCLxTghKJEbBe+7virAg9gQBlTc76XXKKFYRACsET6U8gBvLsGRn0scdKAamTng+Uk19MRcfHpNs7711dsMRof7g3USqVx/X6dDrdNxSZmHtwE9fTOUXdPug06wXLyLA8Xa+qyIyXOC5qKnYaCzv1RDLlub44RZ3aDafWadVaR9twLIMFrN2s2a2jmt3pbMEJGC/ZjpzawWGtfbAFlQjKwyWf4yBho1VrO1uQM2AsmS6RTbvmNNq1dnMLMhQwW6Zut7HEQ/PbAhxTcb0CPmph9k7NdjobL32S+mERZaQwcIZau3MSHClVJ3eXN70W5DNRtKqYQqsMK128O5IwyjRue1DR0iIGNdWXkOEDT8aAeiTd84uXTZSwmjFYu28klAbUhI2Hq90QlMS5wyBQOGWwt4a2i14UtnluxDzLhMn1Cr0o7M3rAy4zAQY5H144JnQvd58UjHtV2xVROFFPP9KfC6DMpF9lETH51X1ysRJxAji8BLxAdy8FQRV23eX8Pc5S/Kt8BNAXiZfrvjCA1ZzzSVxc4Wpe09OR/jCBXRXxGcGLxbWsBOevvTlt7xqvnH27vd/6zak+7y8m8UPrW+s7a8+yrY713Dqz+taV5ZdOSm9LsqTKf5T/Lv9T/ncOvVdaxHxjra3yf/8DoJOLLA==</latexit>

Per-node problem size

Bnet

ops
time

words
time

45TH ASE SEMINAR (2023) — TŌKYŌ

Young & Vuduc (2016). “Finding balance in the post-Moore’s Law era.”Matrix multiplication (same peak op/s, strong scaling, ORNL Summit-like)
Big sockets

Small sockets

Time
Compute2

10

Network

0

10

−2

10

Memory
−4

10

1

10

2

10

10

3

10

4

10

5

1

10

# of sockets

10

2

10

3

10

4

10

5

Matrix multiplication (same peak op/s, strong scaling, ORNL Summit-like)
Big sockets

Small sockets

Time

Tradeoff

Compute2

10

Network

0

10

−2

10

Memory
−4

10

1

10

2

10

10

3

10

4

10

5

1

10

# of sockets

10

2

10

3

10

4

10

5

Matrix multiplication (same peak op/s, strong scaling, ORNL Summit-like)
Big sockets

Small sockets

Time
Compute210~ 2.4x

Network

0

10

−2

10

Memory
−4

10

1

10

2

10

10

3

10

4

10

5

1

10

# of sockets

10

2

10

3

10

4

10

5

3D FFT (same peak op/s, strong scaling, ORNL Summit-like sockets)
Big sockets

Time

Small sockets

5

10

4

10

Network

3

10

2

10

Memory

1

10

0

10

Compute
10

1

2

10

10

3

10

4

10

5

10

1

# of sockets

10

2

3

10

10

4

10

5

Communication
3D FFT (same peak op/s, strong scaling, ORNL Summit-likesockets) tradeoffs
Big sockets

Time

Small sockets

5

10

4

10

Network

3

10

2

10

Memory

1

10

0

10

Compute
10

1

2

10

10

3

10

4

10

5

10

1

# of sockets

10

2

3

10

10

4

10

5

3D FFT (same peak op/s, strong scaling, ORNL Summit-like sockets)
Big sockets

Time

Small sockets

5

10

4

10

Network3

10~ 5.9x2

10Memory

1

10

0

10

Compute
10

1

2

10

10

3

10

4

10

5

10

1

# of sockets

10

2

3

10

10

4

10

5

DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )“ Communication isinevitable & systemcomponents help manage
scalability tradeoffs.
CONCLUSION SO FAR?

37

45TH ASE SEMINAR (2023) — TŌKYŌSmartNICs (DPUs)
as DMXs (data-movement accelerators)

38

45TH ASE SEMINAR (2023) — TÕKYÕDPU-DMXs inmodern clusters
The basic building block of a distributedmemory cluster or supercomputer is a node.
Each node includes a host, which is aprocessor (xPU) + memory hierarchy.
The host can communicate with other hosts
via its NIC (network interface controller).
A network connects the nodes. The nodes
may be arranged in some topology, which
determines the network’s carrying capacity
and cost.
In a DPU, the NIC becomes “host-like” via
the addition of processing (ypu) andmemory.
39

Node

Host
Mem
$
xPU

!NIC
mem
$
ypu

Network

Uses of DPUs,
actual andenvisioned• Accelerating network applications: packet
classification, traffic shaping, multicasting• Network, storage, and sensor algorithms, e.g.,

distributed key-value stores, consensus algorithms,
computer vision• Advanced runtimes: e.g., distributed resource and I/Omanagement, fault tolerance (See Ryan et al. IPDRM’20)

OPPORTUNITIES AND PITFALLS: SEE GRANT ET
AL. IPDRM’20 SURVEY, “RADD RUNTIMES”

40• HPC algorithms?

DPU algorithms• MiniMD ( S. Karamati et al., "“Smarter” NICs for faster molecular dynamics: a case study," IPDPS, 2022 )• Maxwell’s equation (Current work)

41

BlueField-2 DPU - 2x 25Gb/s HHHL
form factor

Our focus• Platform: DPUs like BF2 & BF3, which are based on
general-purpose multicore CPUs

• Usage model: Off-path computation (i.e., asynchronous,

independent progress) rather than on-path (i.e., “on-the-wire”
computation)• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

BlueField-2 DPU - 2x 100Gb/s FHHL
form factorfi

42Our focus
Off-path (async & indep threads)• Platform: DPUs like BF2 & BF3, which are based on
general-purpose multicore CPUs

• Usage model: Off-path computation (i.e., asynchronous,
threads

independent progress) rather than on-path (i.e., “on-the-wire”
computation)

threads• Programming model: Multiprogram MPI with the DPU

On-path (deadline-driven task)

in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

threads

Inline
task

fi

43

Our focus• Platform: DPUs like BF2 & BF3, which are based on

OpenSNAPI

general-purpose multicore CPUs

• OpenSNAPI is a project of the UCF
Consortium• Usage model: Off-path computation (i.e., asynchronous,

independent progress) rather than on-path (i.e., “on-the-wire”
computation)• Straight from the source:– “OpenSNAPI is a collaboration between industry,laboratories and academia with the goal to create astandard application programming interface (API) foraccessing the compute engines on the network, andspecifically on the smart network adapter. OpenSNAPI
allows application developers to leverage the network
compute cores in parallel to the host compute cores foraccelerating application runtime, and to perform operations
and processing closer to the data.”• Programming model: Multiprogram MPI with DPU in“host mode” rather than any vendor-speci c model orlower-level communication library (e.g., OpenSNAPI)

Source: https://www.ucfconsortium.org/projects/opensnapi/

Los Alamos National Laboratory

8/11/20 | 9fi

44Sara Karamati

A MiniMD case study

45
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063“In theory, theory andpractice are the same.
In practice, they are not.”Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
Node
Host
Mem
$
xPU

!NIC
mem
$
ypu

Network46

45TH ASE SEMINAR

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
One host xPU (16 cores)
Host

Mem
BlueField-2
mem

$

$
ypu

xPU

47

45TH ASE SEMINAR (2023) — TÕKYÕ

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
One host xPU (16 cores)
Host

Mem
BlueField-2
mem

$

$
ypu

xPU

657 GF/s (fp64)
48

45TH ASE SEMINAR (2023) — TÕKYÕ

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
One host xPU (16 cores)
Host

Mem
BlueField-2
mem

$

$
ypu

xPU

657 GF/s (fp64)
76.8 GB/s
49

45TH ASE SEMINAR (2023) — TÕKYÕ

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
Host
One host xPU (16 cores)

Host

BF-2 yPUs (no host)Mem
BlueField-2

Mem

mem

$

$
ypu

$
xPU

xPU

657 GF/s (fp64)
76.8 GB/s
50

45TH ASE SEMINAR (2023) — TÕKYÕ

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Sara Karamati
DATA - M O V E M E N T A C C E L E R ATO R S ( D M X S )Target platform: NVIDIA (née Mellanox) BF2
Host
One host xPU (16 cores)

Host

BF-2 yPUs (no host)Mem
BlueField-2

Mem

mem

$

$
ypu

$
xPU

xPU

657 GF/s (fp64)
76.8 GB/s
51

80 GF/s
25.6 GB/s

45TH ASE SEMINAR (2023) — TÕKYÕ

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Baseline MiniMD
MiniMD is a molecular dynamics proxy-app.
It calculates the position and velocity of a setof interacting particles in discrete time steps(iterations).

52
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Proc 1

Proc 2

Baseline MiniMD
MiniMD is a molecular dynamics proxy-app.
It calculates the position and velocity of a setof interacting particles in discrete time steps(iterations).
In the distributed-memory setting, thesimulation domain is divided spatiallyamong MPI processes.
Every process owns its particles, computes
force on these particles and then updates
the position and velocity of these particles.

Proc 3

Proc 453
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Short-range forces

Proc 1Baseline MiniMD

Proc 2

In each iteration, every particle interacts withothers that lie within a some cutoff distance(rc). A particle’s neighbor list storesreferences to them.

rc

Recall:<latexit sha1_base64="wlvbpO/iCFawUxEqvXghJqsRc90=">AAAGhnicdVTdbts2FFbb1cu8v7a73A07o0AKqLElx3aSq2zulgxrZ89t2gKWl9H0kSyEogSSiusSeoA9zW63R9nb7FC2MStxCUggeb7znV+eacZjpVutf+/cvffJ/dqne5/VP//iy6++fvDw0RuV5pLBBUt5Kt9NqQIeC7jQsebwLpNAkymHt9OrvpW/vQap4lS81ssMJgmNRBzGjGq8unzQCKZxNDC//u4XJOCpiGQczTWVMl2QtahAVOugVS5ye+OtNw1nvYaXD+//EsxSlicgNONUqbHXyvTEUKljxqGoB7mCjLIrGoGhiVLLZFqQJwnVc3VTZi93yca5Do8mJhZZrkEwhKAszDnRKbGBklksgWm+xA1lMkbLhM2ppExjOur1J+QlvQJyDvwaUEaJnqMOhDTnumJHMcphNjHzEom+SxCwYGmSUDELQprEfLnWM4EK19uCVEheexMTpqL01Nr+UVCsEJHAsQ7XQFT8IRZRJXaU4S2U8O9znWIGMIRYKJC2dCQNCbzXkhKFGlDN2/vyrlR9DiG2Bjkf9skZRh8BKZuGKDaHBMizZ2Q8xAC1BtIkAwZUkJ+sRycnpD94MbgY8dT2z2R/rnWmTprNxWJxYClyuZLgIWlmK4qmd9hrN0uWS8vytOpVaRkzOCt9Kk8GHTubYreJwozOfiiM1/LdXsftHO/C8RzWsG7b9TrHrtfr7cBJmG3Yjn338MjtHu5ApZKKaMPn+0jY6rhdfwdyCZyniw2y7bl+q+t22zuQkYTlxnS3iy4e2W8HcEbl1Rb4uIPWe67n23D+by8TzDMWlVq2Fcb+xJhgRdLwioZfBJtKV5QYl2Wqxth1FYatLKL+bS2buN1KZUpLHeypoYIc33c6A+xH0n8xetXGFtZLDpV6I6GyoDbceDUmiECrwgQcwtWsKWn7KMXGtk+d2JGQclX10M6jlX8gVC7BIk1g//hGzaAIHpeM+w0vKMfY04/kZwSUW/PbLDIhPwePR1sazwFnl4SXKB5kIKnGrAdCfCiM/dU/AhjKdFqYobSAbZvXVL7Kp2UJt+3anF6aVRAyMYGOxZJgYXFtPMHx690ctrc3b/wDr3vQ+e2wcfrHehDvOd863zn7juf0nFPn3Bk6Fw5z/nT+cv52/qnt1Q5qnVpvBb17Z63zjVNZtdP/AKYCPQM=</latexit>

O N
Proc 3

Proc 454
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

2! O(N )

Proc 1

Proc 2

Baseline MiniMD

Δ
In each iteration, every particle interacts withothers that lie within a some cutoff distance(rc). A particle’s neighbor list storesreferences to them.

rc

The neighbor list must be updated as
particles move. But such updates are
expensive! So every list includes a buffer of“extra” particles that lie within a surrounding
annulus, or “skin,” parameterized by its
thickness (Δ).

Proc 3

Proc 455
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Proc 1

Proc 2“Ghost zone”(comm+overhead)

Baseline MiniMD
The cutoff distance (rc) and skin thickness (Δ)
imply the size of the interaction region justoutside the boundaries of each process.
Each process keeps a copy of particles inthat region.

rc Δ

Proc 3

Proc 4

56
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Proc 1

Particles move through subdomains(comm+overhead) Proc 2

Baseline MiniMD
Particles are reassigned to new processes as
they move through the spatial domain.
Neighbor list updates, boundary region
exchanges, and particles reassignment toprocesses are triggered every so often via auser-selected parameter (e.g., every kiterations).

Δ

Proc 3

Proc 457
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Update velocity (UV)

Baseline MiniMD
Proc 1

Update position (UX)

Neighborbuild iteration?

Proc 2No

Yes

Exchange (EX)Communicate (C)Update border lists (UB)
Build neighbor lists (NB)

Compute force (FC)Proc 3

Proc 4

Update velocity (UV)

Each task is parallelizable but the sequence is sequential as shown by edges
58
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Serial dependencies

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

59
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

Serial dependencies

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

60
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

Serial dependencies
Computational work
(force computation)

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

61
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

FC work: force computation

Serial dependencies
Computational work(velocity update)

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

62
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

FC work: force computation
UV work: update velocity

Serial dependencies
Data reorg[neighbor-list (re)build]

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

63
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

FC work: force computation
UV work: update velocity

Serial dependencies

Neighbor-list
Generation
Host

Iteration

NB neighbor-list rebuild

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

64
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

FC work: force computation
UV work: update velocity

Serial dependencies

NB neighbor-list rebuildComm[ghost zone communication]

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

65
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

FC work: force computationC

communication

UV work: update velocity

Serial dependencies

NB neighbor-list rebuild

Data reorg + comm[particles reallocation and border lists update]

Neighbor-list
Generation
Host

Iteration

k

k+1

EX UB

i

NB

FC

UV

C

FC

i+1

UV

C

k+2
FC

i+2

66
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

Breaking the dependencies

FC work: force computationC

communication

UV work: update velocity

UB comm: update border lists

NB neighbor-list rebuild

EX comm: particles reallocation(“Off-path” algorithm)

Neighbor-list
Generation
Bluefield
Host

Iteration

k

k+1

k+2

FC
EX UB

i

FC
NB

FC

UV

C

FC

i+1

UV

C

FC

i+2

67
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

NB

i+3

FC

UV

i+4

Breaking the dependencies

FC work: force computationC

communication

UV work: update velocity

UB comm: update border lists

NB neighbor-list rebuild

EX comm: particles reallocation(“Off-path” algorithm)

Neighbor-list
Generation
Bluefield
Host

Iteration

k+1

k

k+2

FC
EX UB

i

FC
NB

UV

C

FC

i+1

UV

C

FC

i+2

68
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

EX UB

i+3

NB

UV

i+4

Breaking the dependencies

FC work: force computationC

communication

UV work: update velocity

UB comm: update border lists

NB neighbor-list rebuild

EX comm: particles reallocation(“Off-path” algorithm)

Neighbor-list
Generation

Iteration

k+1

kBluefield

C

FC

Host

EX UB

i

k+2

EX
NB

C
UV

C

FC

i+1

UV

C

FC

i+2

69
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

UV

FC

EX UB

i+3

EX
NB

UV

i+4

Baseline
experiments• System: 16 nodes, In niband HDR (100 Gbps)• Hosts: (2-socket) x (16-core Intel Broadwell E5-2697A,
2.6 GHz) + (256 GiB DDR4 RAM, 2400 MHz)

• NICs per node
• 1 x NVIDIA ConnectX-6 HDR100 (100 Gbps)
In niBand/VPI adapters• 1 x NVIDIA BlueField-2 SoC — (8-core ARMv8 A72,
“THOR” CLUSTER, MAINTAINED BY THE HPC·AI
ADVISORY COUNCIL [LINK]

2.5 GHz) + (16 GiB DDR4 RAM) + (HDR100)

70

fi

fi

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Restructured
method …

Re-neighboring
Interval
Higher is better71
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Restructured
method is faster
We observe small, but largely uniform,
speedups of up to 20% compared to hostonly execution with conventional NICs.
This improvement compares favorablywith the power increase on each node due
to BF2, which we estimate from sensors tobe as little as 6%.

72
S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

Re-neighboring
Interval

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

Hypothetical: Multi-SmartNIC
a.k.a., revisiting the “iron law”

Host
One host xPU (16 cores)

Host

BF-2 yPUs (no host)Mem
BlueField-2

Mem

mem

BlueField-2
mem
$

$
ypu

$

$

xPU
ypu

xPU

657 GF/s (fp64)
76.8 GB/s
73

80 GF/s
25.6 GB/s

K A R A M AT I E T A L . , I P D P S ’ 2 2

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

Hypothetical: Multi-SmartNIC
a.k.a., revisiting the “iron law”

Host
One host xPU (16 cores)

Host

Host

Mem

Mem

Host

$

Host

BlueField-2

657 GF/s (fp64)
76.8 GB/s

mem

$

$

Host

ypu

Host

ypu

Mem

Mem

Mem

xPU

xPU

BlueField-2
xPU

BlueField-2
xPU

$

$

mem

$
ypu

xPU

xPU

mem

$

$

mem

mem

$

ypu

ypu

BlueField-2

BlueField-2

mem

mem

$
ypu

640 GF/s
204 GB/sxPU(aggregate)

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

BlueField-2

$

$

ypu

xPU

BlueField-2

$

$

Mem

xPU

K A R A M AT I E T A L . , I P D P S ’ 2 2

Mem

mem

$

74

Mem

BlueField-2Mem$

8 xHostBF-2 yPUs (no host)

Host

$
ypu

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

Hypothetical: Multi-SmartNIC
a.k.a., revisiting the “iron law”

One host xPU (16 cores)
Host

Mem

8 x BF-2 yPUs (no host)
BlueField-2
mem

BlueField-2
mem$

BlueField-2
mem$

BlueField-2
mem

$

$

ypu

ypu

ypu

ypu

BlueField-2

BlueField-2

BlueField-2

BlueField-2

$
mem

mem

$

xPU

~ 8.5 F:B
75

ypu

mem

$
ypu

$
ypu

~ 3.1 F:BK A R A M AT I E T A L . , I P D P S ’ 2 2

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

mem

$
ypu

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

Hypothetical: Multi-SmartNIC
a.k.a., revisiting the “iron law”

One host xPU (16 cores)
Host

Mem

8 x BF-2 yPUs (no host)
BlueField-2
mem

BlueField-2
mem$

BlueField-2
mem$

BlueField-2
mem

$

$

ypu

ypu

ypu

ypu

BlueField-2

BlueField-2

BlueField-2

BlueField-2

$
mem

mem

$

xPU

76

ypu

mem

$
ypu

mem

$
ypu

$
ypu

Time = “1”Speedup ~ 1.7x

using all cores

Real measurement on MiniMD!

K A R A M AT I E T A L . , I P D P S ’ 2 2(Similar for P3DFFT, SuperLU_DIST)

S. Karamati (GT) et al. (Sandia, Queens U.; 2022) — “Smarter NICs for faster molecular dynamics: a case study.” doi:10.1109/IPDPS53621.2022.00063

miniEM (MueLu) —Maxwell solver case study

77Baseline Maxwell
SolverMaxwell multigrid solver from MueLu, an open-source software
library within the Trilinos project

Setup Phase

Solve Phaseũ ← PreFineRelaxation()r̃ ← r − ( DT1 M2 D1 + M1 ) ũ

P1 =FormSpecialProlongator()

Form AH ← P1T (M1D∗1D1 + M1D0D∗0 + M1)P1

Standard AMG Setup(AH)Standard AMG Setup(A22)

Bochev, Pavel B., et al. "An algebraic multigrid approach based on a compatible gauge
reformulation of Maxwell's equations." SIAM Journal on Scientific Computing 31.1 (2008)a ← Standard AMG Vcycle(AH)p ← Standard AMG Vcycle(A22)ũ ← PreFineRelaxation()ũ ← PreFineRelaxation()Baseline Maxwell
SolverSolve Phaseũ ← PreFineRelaxation()Standard AMG Vcycle(AH) and Standard AMG
Vcycle(A22) operate independently and can be
executed in parallel for optimized performance.

r̃ ← r − ( DT1 M2 D1 + M1 ) ũa ← Standard AMG Vcycle(AH)p ← Standard AMG Vcycle(A22)ũ ← PreFineRelaxation()ũ ← PreFineRelaxation()Proposed Parallel
Execution of AMG Cycles
DPU

Host

Solve Phaseũ ← PreFineRelaxation()r̃ ← r − ( DT1 M2 D1 + M1 ) ũa ← Standard AMG Vcycle(AH)p ← Standard AMG Vcycle(A22)ũ ← PreFineRelaxation()ũ ← PreFineRelaxation()AMG V-Cycles
Multigrid Methods Involve:
1. Smoothing:• Utilizes simple iterative methods like GaussSeidel.
• Reduces oscillatory high-frequency error.

2. Coarse-grid Correction:• Transfers information to a coarser grid throughrestriction.
• Solves the coarse-grid system of equations.
• Eliminates low-frequency error.

3. Interpolation:• Transfers the solution back to the ne grid.

Source: Multi grid V-cyclefiComputational costs are primarily governed by sparse matrix operations

fl

fl

fl

fi

Experimental Testbed
HostBlue eld-3

CoreIntel Broadwell E5-2697A

Arm Cortex A-78

# Sockets

2

1

Cores/Socket

16

16

Clock (GHz)

2.6

2.25

Private L1 DCache (per core)

32 KB

64KB

Private L2 Dcache (per core)

256 KB

512 KB

Shared L3 Cache (per node)

80 MB

16 MB

DRAM

DDR4 (4800 MT/s)

DDR5 (5600 MT/s)

Peak op/s per socket (FP64)

656.6 G op/s

288 G op/s

Peak GB/s per socket

76.8 GB/s

69.21 GB/s

Matrix Representation and
Non-Zero Element Distribution

AH

Matrix

rows

nnz

AH

4,065

346,665

A22

17,261

248,581

A22

ffl

Observation: A22 is smaller but AH stresses cache more, so it’s not clear a priori which to o oadComparative Benchmarking:“TUNED" STREAM benchmark on Host vs. Blue eld
Evaluating with Array of 20,000,000 elements, running each test 100 times.
“TUNED" STREAM benchmark on Blue eld-3
100000

75000

75000

50000

50000

25000

25000

0

0

Copy

1 OMP Thread
8 OMP Threads

Scale

Add

2 OMP Threads
16 OMP Threads

Triad

4 OMP Threads
32 OMP Threads

Copy

1 OMP Thread
8 OMP Threads
fi

MB/s

100000fi

MB/s“TUNED" STREAM benchmark on Host

Scale

Add

2 OMP Threads
16 OMP Threads

Triad

4 OMP Threads

Comparative Benchmarking:Sparse Matrix Vector Performance on Host vs. Blue eld
Evaluating with Matrix of 163,617 Rows and 13,662,045 nnz Elements
The execution of SPMV (MPI)

4

4

3

3

2

2

1

1

0

0

Host

1 OMP Thread
8 OMP Threads

BF3

2 OMP Threads
16 OMP Threads

4 OMP Threads
32 OMP Threads

Host

1 MPI Process
8 MPI ProcessesfiTime (sec)Time (sec)The execution of SPMV (OpenMP)

BF3

2 MPI Processes
16 MPI Processes

4 MPI Processes
32 MPI Processes

Preliminary results: End-to-End Solve-Phase Timesfor miniEM: Host vs. Host + BlueField3
Times shown are end-to-end solver times.

Problem Size

#Host Cores / #BF3 Host-only Time(seconds)
Cores

H+BF Time(seconds)

Relative
Scaling120

256 / 12885.35876.11031.12

120

128 / 64137.062129.9281.05

120

64 / 32220.767

213.9191.03

80

256 / 12859.86250.94771.17

80

128 / 6472.0195

63.54821.13

8064 / 32

95.3597

87.6009

1.09

60

256 / 12838.5208

31.2498

1.23

60

128 / 64

42.61937.58461.13

60

64 /3251.4106

46.4185

1.11

Summary
Communication is fundamental andinevitable, so anything that addresses
it should be pursued vigorously.
Restructuring algorithms, especially
increasing asynchrony, can exploit
smartNICs in HPC. We are pursuing avariety of candidates, including
distributed time-tiled stencils, AMR,
novel collectives, among others.
Many open questions remain,
regarding other techniques,programming, runtimes, andperformance modeling.
87

Node
Host
Mem

💡NIC

$

$

xPU

mem

cpu

Asynchronous I/O in BlueField Target using OpenMP#pragma omp parallel#pragma omp single{#pragma omp task
#pragma omp target nowait{for (i = 0; i < 5; ++i)
printf(“hola - %05d\n”, i);} // end omp target#pragma omp task{for (j = 0; j < 5; ++j)
printf(“adios - %06d\n”, j);} // end omp task
} // end omp single

102

Asynchronous I/O in BlueField Target using OpenMP#pragma omp parallel#pragma omp single{#pragma omp task
#pragma omp target nowait{for (i = 0; i < 5; ++i)
printf(“hola - %05d\n”, i);} // end omp target#pragma omp task{for (j = 0; j < 5; ++j)
printf(“adios - %06d\n”, j);} // end omp task
} // end omp single

102

Asynchronous I/O in BlueField Target using OpenMP#pragma omp parallel#pragma omp single{#pragma omp task
#pragma omp target nowait{for (i = 0; i < 5; ++i)
printf(“hola - %05d\n”, i);} // end omp target#pragma omp task{for (j = 0; j < 5; ++j)
printf(“adios - %06d\n”, j);} // end omp task
} // end omp single

102Summary
Communication is fundamental andinevitable, so anything that addresses
it should be pursued vigorously.
Restructuring algorithms, especially
increasing asynchrony, can exploit
smartNICs in HPC. We are pursuing avariety of candidates, including
distributed time-tiled stencils, AMR,
novel collectives, among others.
Many open questions remain,
regarding other techniques,programming, runtimes, andperformance modeling.
91

Node
Host
Mem

💡NIC

$

$

xPU

mem

cpu

Director’s cut

92

45TH ASE SEMINAR (2023) — TÕKYÕ

T H E F U T U R E I S S PA R S E @ S C 2 3

Four “generations” ofcomputing

Gregory Abowd (2016). “Beyond Weiser: From ubiquitous
computing to collective computing.” DOI: 10.1109/MC.2016.22

OUTLOOK
TABLE 1. A framework for comparing computing generations, inspired by Mark Weiser.
Application

Generation

Time frame

Human–computer
ratio

1

Mid-1930s

Many–1

Mainframe

Scientific calculation

Data processing

2

Late 1960s

1–1

PC

Spreadsheet

Database management,
document processing

3

Late 1980s

1–manyInch/foot/yard

Calendar and contact
management, human–
human communication

Location-based services,
social media, app ecosystem,
education4

Mid-2000s

Many–many

Cloud/crowd/shroud

Personal navigation andentertainment

Health advisors, educational
assistants, supply chain logistics

Canonical device

InitialFollow-on

T H E F U T U R E I S S PA R S E @ S C 2 3

Four “generations” ofcomputing

Gregory Abowd (2016). “Beyond Weiser: From ubiquitous
computing to collective computing.” DOI: 10.1109/MC.2016.22

OUTLOOK
TABLE 1. A framework for comparing computing generations, inspired by Mark Weiser.
Application

Generation

Time frame

Human–computer
ratio

1

Mid-1930s

Many–1

Mainframe

Scientific calculation

Data processing

2

Late 1960s

1–1

PC

Spreadsheet

Database management,
document processing

3

Late 1980s

1–manyInch/foot/yard

Calendar and contact
management, human–
human communication

Location-based services,
social media, app ecosystem,
education4

Mid-2000s

Many–many

Cloud/crowd/shroud

Personal navigation andentertainment

Health advisors, educational
assistants, supply chain logistics

Canonical device

InitialFollow-on

T H E F U T U R E I S S PA R S E @ S C 2 3

Four “generations” ofcomputing

Gregory Abowd (2016). “Beyond Weiser: From ubiquitous
computing to collective computing.” DOI: 10.1109/MC.2016.22

OUTLOOK
TABLE 1. A framework for comparing computing generations, inspired by Mark Weiser.
Application

Generation

Time frame

Human–computer
ratio

1

Mid-1930s

Many–1

Mainframe

Scientific calculation

Data processing

2

Late 1960s

1–1

PC

Spreadsheet

Database management,
document processing

3

Late 1980s

1–manyInch/foot/yard

Calendar and contact
management, human–
human communication

Location-based services,
social media, app ecosystem,
education4

Mid-2000s

Many–many

Cloud/crowd/shroud

Personal navigation andentertainment

Health advisors, educational
assistants, supply chain logistics

Canonical device

InitialFollow-on

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on generalpurpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• Applications: HPC algorithms and proxy-apps withaggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU

SEE PART 3 OF THIS TALKfi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl94

in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)BlueField-2 DPU - 2x 25Gb/s HHHL
form factor
SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on generalpurpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,

independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• Applications: HPC algorithms and proxy-apps withaggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)BlueField-2
DPU- 2x 100Gb/s FHHL
SEE PART 3 OF THIS
TALK
form factorfi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl95• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on generalpurpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• Applications: HPC algorithms and proxy-apps withaggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

fi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl96SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on general-

Off-path (async & indep threads)

purpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• Applications: HPC algorithms and proxy-apps withaggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

fi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl96SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on general-

Off-path (async & indep threads)

purpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)

threads• Applications: HPC algorithms and proxy-apps withthreads

aggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

fi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl96SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on general-

Off-path (async & indep threads)

purpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)

threads• Applications: HPC algorithms and proxy-apps withthreads

aggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes

On-path (deadline-driven task)(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)

fi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl96SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on general-

Off-path (async & indep threads)

purpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,
independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)

threads• Applications: HPC algorithms and proxy-apps withthreads

aggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes

On-path (deadline-driven task)(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with the DPU

threads

Inlinetaskfi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl96

in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)BluesMPI: MPI Non-blocking Alltoall Oﬄoading on BlueField Smart NICs

25

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Step 1) Metadata Aggregation to the Host Communicator Leader Process:Once a process in the host communicator registers its send and receive buﬀers,
it creates a collective info object that includes RDMA buﬀer addresses andkeys. It also includes this process’s rank in MPI COMM WORLD as well as thecount and datatype of this collective call. This information is the Metadata forthe collective call from this host process. The host communicator leader (whichis rank 0 in our design) gathers the Metadata from all the processes in thecommunicator.

Our foci (gaps)• Platform: DPUs like BF2, which are based on generalpurpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,

independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• Applications: HPC algorithms and proxy-apps withaggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)Fig. 4. BluesMPI procedure to oﬄoad non-blocking Alltoall collective operation to theworker processes on the Smart NIC. Step 0 is not included in this figure.

fi

flStep 2) Metadata Registration with HCA and Oﬄoading the Task Object to
Leader of the Workers Group: Once the host communicator leader generates thearray of Metadata, it has to register this array with HCA so that all the Worker
SEE PART 3 OF THIS TALK
process on the Smart NIC can read whatever information that they require atany time during progressing the collective. Once the registration is done, the hostcommunicator leader creates a new task object and sends it to the Workers groupleader. This task object has the information about the type of the collective andthe algorithm which must be performed by the Worker processes on the Smart
NIC. It also has the RDMA information of the Metadata array and the hostcommunicator size.
From
toN Operform
P A Rprocesses
T 1 : S M A R TareN I CfreeTECH
L O G I E S useful application compu97now, the hosttation. In the meantime, the leader of the Worker group on Smart NIC waits
for the incoming task objects from the leaders of the host communicators. Since
the application could have several sub-communicators, the leader of the Work-• Programming model: Multiprogram MPI with the DPU
in “host mode” rather than any vendor-speci c modelor lower-level communication library (e.g., OpenSNAPI)SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Our foci (gaps)• Platform: DPUs like BF2, which are based on generalpurpose multicore CPUs (e.g., generalizing INCA)• Usage model: Off-path computation (i.e., asynchronous,

OpenSNAPI

independent progress) rather than on-path (i.e., “on-the-wire”
computation, e.g., sPIN)• OpenSNAPI is a project of the UCF
Consortium• Applications: HPC algorithms and proxy-apps with• Straight from the source:

aggressive restructuring rather than relying on
middleware, “basic” porting, or simple of oad schemes– “OpenSNAPI is a collaboration between industry,laboratories and academia with the goal to create astandard application programming interface (API) foraccessing the compute engines on the network, andspecifically on the smart network adapter. OpenSNAPI
allows application developers to leverage the network
compute cores in parallel to the host compute cores foraccelerating application runtime, and to perform operations
and processing closer to the data.”(e.g., BluesMPI, Williams et al. PENNANT study, which found no
speedup)• Programming model: Multiprogram MPI with DPU in

SEE PART 3 OF THIS TALK

Source: https://www.ucfconsortium.org/projects/opensnapi/

Los Alamos National Laboratoryfi

PA R T 1 : S M A R T N I C T E C H N O LO G I E S

fl98

8/11/20 | 9“host mode” rather than any vendor-speci c model orlower-level communication library (e.g., OpenSNAPI)

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

Anatomy of asupercomputer
The basic building block of a distributedmemory cluster or supercomputer is a node.
Each node includes a host, which is aprocessor (xPU) + memory hierarchy.
The host can communicate with other hosts
via its NIC (network interface controller).
A network connects the nodes. The nodes
may be arranged in some topology, which
determines the network’s carrying capacity
and cost.

99

Node

Host
Mem

NIC

$
xPU

Network

“ S M A R T E R ” N I C S F O R FA S T E R M O L E C U L A R DY N A M I C S : A C A S E S T U DY

DPUs in modern
clusters
The basic building block of a distributedmemory cluster or supercomputer is a node.
Each node includes a host, which is aprocessor (xPU) + memory hierarchy.
The host can communicate with other hosts
via its NIC (network interface controller).
A network connects the nodes. The nodes
may be arranged in some topology, which
determines the network’s carrying capacity
and cost.
In a smartNIC, the NIC becomes “host-like”
via the addition of processing (ypu) andmemory.
100

Node

Host
Mem
$
xPU

!NIC
mem
$
ypu

Network

Hybrid MPI/OpenMP
performance results

Lower is better101Hybrid MPI/OpenMP
performance results
Our algorithm works best when it cancompletely hide the force computation timeon BlueField.
The degree of achievable overlap depends
on the relative computational power of thehost and BlueField.
The knee of each curve indicates where therunning times of neighbor-build on the hostand force-compute on the BlueField areclosest.
Thread synchronization overhead in the force
computation routine causes the performancenot to scale proportionally to the number of
threads.
102

Hybrid MPI/OpenMP
performance results
Our algorithm works best when it cancompletely hide the force computation timeon BlueField.
The degree of achievable overlap depends
on the relative computational power of thehost and BlueField.
The knee of each curve indicates where therunning times of neighbor-build on the hostand force-compute on the BlueField areclosest.
Thread synchronization overhead in the force
computation routine causes the performancenot to scale proportionally to the number of
threads.
103Hybrid MPI/OpenMP
performance results
Our algorithm works best when it cancompletely hide the force computation timeon BlueField.
The degree of achievable overlap depends
on the relative computational power of thehost and BlueField.
The knee of each curve indicates where therunning times of neighbor-build on the hostand force-compute on the BlueField areclosest.
Thread synchronization overhead in the force
computation routine causes the performancenot to scale proportionally to the number of
threads.
104

1core

2.5x faster8 cores

An explanatory
performance modelttotal = ( tN + tFC) × #re-neighboring + tremain

Host

EX UB

NB

FC

UV

C

FC

UV

ttotal
105

C

FC

UV

EX UB

NB

FC

UV

An explanatory
performance modelttotal = ( tN + tFC) × #re-neighboring + tremain

Host

EX UB

NB

FC

UV

C

FC

UV

ttotal
106

C

FC

UV

EX UB

NB

FC

UV

An explanatory
performance modelttotal = ( tN + tFC) × #re-neighboring + tremain

Host

EX UB

NB

tN

FC

UV

C

FC

UV

tFC

FC

UV

EX UB

NB

tN

ttotal
107

C

FC

tFC

UV

An explanatory
performance modeltF = tEX + tFC + tC
tEX
Bluefiled

108

EX UB

tC

tFC
NB

FC

UV

C

FC

UV

C

FC

UV

EX UB

NB

FC

UV

An explanatory
performance model
toff-path = tremain + (ax(tN , tF) × #re-neighboring
Bluefield

C

Host

EX UB

FC
NB

EX

C
UV

C

FC

UV

toff-path
109

C

FC

UV

EX UB

FC
NB

EX
UV

An explanatory
performance model
toff-path = tremain + (ax(tN , tF) × #re-neighboring
Bluefield

C

Host

EX UB

FC
NB

EX

C
UV

C

FC

UV

toff-path
110

C

FC

UV

EX UB

FC
NB

EX
UV

An explanatory
performance model
toff-path = tremain + max(tN , tF) × #re-neighboringtF
Bluefield

C

Host

EX UB

FC

EX

NB

C
UV

C

FC

UV

C

FC

UV

EX UB

FC

EX

NB

tN
toverlap = max(tN , tF)

toverlap = max(tN , tF)

toff-path111

UV

Predictive power of ourperformance model
The model can closely predict the algorithm
runtime.

Actual Measurements

4 cores on BlueField

8 cores on BlueField

112

Predicted by Model

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

113

$

xPU

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

114

$

xPU

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

80 GF/s peak (8 cores)
115

$

xPU

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

80 GF/s peak (8 cores)
116

$

xPU

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

80 GF/s peak (8 cores)
117

$

xPU

40 GF/s peak (1 core)

How about a DPU with more
performant core?
Actual system
Host

Hypothetical (modeled)
Host

Mem

Mem

BlueField-2

Auxiliary node

mem

$

$

mem

$
ypu

ypu

xPU

80 GF/s peak (8 cores)
118

$

xPU

40 GF/s peak (1 core)Same predicted performance!
(due to no sync overhead on aux node)Hypothetical (modeled)
Host

Mem
Auxiliary node
mem

$

$
ypu

xPU

40 GF/s peak (1 core)119

80 GF/s peak (8 cores)

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

Other highlights fromthe paper
Conducted performance analysis using the
OSU Microbenchmarks suite and the “offthe-shelf” version of MiniMD to understandthe opportunities and limitations presented
by the BlueField for potential HPC
applications.

TABLE IE XPE RIM ENTAL S YST EM C ONFIGURAT ION . T HE TES TBE D IS A
32- NOD E CLU STE R , WHE RE EAC H
B LUE F IELD . E ACH ROW OF THE TABNOD E CON TAIN S A DUA L - SOC
LE BEL OW IS A PER - NOD E CON
KET X 86 HOS T AND ONE
FIGU RAT ION . T HE LINK BAN
DWI DTH IS 12.5 GB/s (I NFIN
I BAND HDR AT 100 Gbps).

Host

Sockets x CPU

Thor
ThorBF

2 ⇥ Intel Broadwell (E5-2697A), 2.6
GHz
1 ⇥ Arm A72, 2.5 GHz

Cores
per socket

Peak flop/s
per socket

Memory168656.6 Gflop/s
80.0 Gflop/s

Peak GB/s
per socket

256 GiB
16 GiB

Device Type76.8 GB/s
25.6 GB/s

Host CPU
BlueField P-Series“Smarter” NICs for faster molecular dynamics:
a case study(a) OSU MPI latency(b) OSU MPI bandwidthFig. 3. OSU MPI latency and band
width tests, relative to conventional
host-to-host communication: BF-to-BF
host-to-host communication for mes
latency is higher, and bandwidth lowe
sage sizes under 16 KiB. (Data sizesr, thanare in bytes.) †⇤†‡†Sara Karamati , Clayton Hughes , K. Scott Hemmert , Ryan E. Grant , W. Whit Schonbein ,Scott Levy† , Thomas M. Conte⇤ , Jeffrey Young⇤ , Richard W. Vuduc⇤⇤ Georgia Institute of Technology, Atlanta, Georgia, USA† Sandia National Laboratories, Albuquerque, New Mexico, USA

Veri ed that the computed simulation results
for the restructured method are still within an
acceptable level of accuracy, in terms of
calculated physical quantities.

‡ Queen’s University, Kingston, Ontario, Canada⇤ Email: {s.karamati,jyoung9,richie}@gatech.edu

Abstract—This work evaluates the benefits of using a “smart”
We consider this question using the case study of MiniMD,
network interface card (SmartNIC) as a compute accelerator a molecular dynamics (MD) simulation benchmark. MiniMD
for the example of the MiniMD molecular dynamics proxy is simple enough to study in detail while also having chalapplication. The accelerator is NVIDIA’s BlueField-2 card, which
includes an 8-core Arm processor along with a small amount of lenging characteristics: the simulation behavior and its communication to computation ratio can vary with different input
OSU
DRAM and storage. We (a)testtheMPInetworking
multiple message andrate data move(b) OSU MPI multiple bandwidthFig. 4. OSU
ment performanceof MPIthese
compared
toiplea band
standard
Intel and configuration-parametervalues, including problem size,
multcards
iple message
rate and multwidth tests, run between 8 pairs ofaround 16 KiB occurs when BF-to-BF
BF-t
o-BF
or
host
-to-host proc
communserver host using microbenchmarks andMiniMD.
In rmsMiniMD,
ication outperfo
esses:theLike re-neighboring
Fig. 3, a crossovertheprocessors,andfrequency,
host-to-host com
munnumber
ication. (Dataof
sizes are in bytes.)
we identify two distinct classes of computation, namely core among others, making
it an excellent choice to examine thecomputation andmaintenancecomputation, which are executed
The exe
cution time bre
akdown of MiniMD, in the host-onlyin some detail.
comhardwarein sequence.settWe
thealgorithm and code to weaken thenew
munication routines to BlueField
ing,restructureappears in Fig. 7. Here, ttotal is the overall executi
wouto
ld notresult MiniMD to
threekeyancchallenges
porting
onit in a Theresignificantare
this dependencetaskparallelism,
therebymakingovetime, andrall
pertforce increase
form
is the tim
egain. Additionally, sincee consumed by the force_comput
e() Min
iMD’s commu
BlueField
profitably.
First,
thereis ar comsequential dependence
possible to rou
increaseof the BlueField-2 concurrently
nication tasktine, tneigutilizations depend
on prioh is the time consumed for the neighbputation
or_
bui
ld()
step
s,
dec
oup
ling
with the host.
Weevaluate
ourimplementationonaclusterthesroutine, and tcomm is cumulative tim
e
rou
tine
between
the
core
parts
of
each
iteration
oftionMiniMD, maks
from
the
rest
of the applicae spent on the exchange(), and offl
oad
ing
them
consisting ofbor
16derdual-socketIntel
Broadwell
host
nodes
with
one
to
a
(), and commu
copro
cessor, whioffload
nicate() routines. The time tle achievicommunicationing
it difficult to concurrentlyorng full
com
com
m is
putation-communication overlap, is
BlueField-2not
perpurhost-node.
Our
results
show
that
while
the
overalle communica
not
tion
a
time; it also includes the time retrivial taskSecond,.
routines to BlueField.
for larger
quired to prepar
compute performanceof eBlueField-2
limited,using them with computation-related
the data for iscom
So
wha
t
cou
ld
municationbedone. We can see
inst
ead
?
Fig
ure
7 indicatessizes, MiniMD’s communication
timethat is small comthat tcomm algorithm
a modified MiniMD
fortheupove
to rall
20%exespeedup over addproblem
has a smallallows
itional computation overlap may beshare of
cuti
on
tim
e.
pos
sibl
e.
Thi
s
find
ing
Therefore, in awith
to itscomputation
the host CPU
baseline
nolueF
loss
simulation
accuracy.
host-B
moparedtivates ourieldinhyb
designrid setting, offl
approach time, making it nontrivial to extractoad
ing only

I. I NTRODUCTION

fi

120

SmartNICs are network interface cards with extended com-5
putational capabilities (see Section II-A and the survey by
Grant et al. [1]). There is growing agreement that the ability
to do on-NIC compute will have a critical enabling rolein cloud and datacenter architectures, especially in tacklingin Section IV, which seeks toofflreasonable
oad work to Blu
performanceby hiding communication load. Third,
eField.

our experiments with the OSU MPI microbenchmark in Section III show that BlueField does not actually outperform conventional host-to-host communication in latency or bandwidth
due to its slow CPU.
To overcome these challenges, our strategy is to change
the algorithm. The original MiniMD algorithm relies on an
expert-tuned parameter-based heuristic to periodically rebuild

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
(Blue line) For message sizes under 16 KiB,
BF2-to-BF2 communication is slower thanhost-to-host communication using
conventional In niband NICs.
(Orange line) For messages under 64 KiB, it
is even slower to exchange messages
between the host and the BF2 on the same
node!
Similar ndings hold for multi-paircommunication and all-gather operations.
Thus, our best bet for getting any
performance improvement will be via offpath execution.

PA R T 3 : M I N I M D C A S E S T U DY

fi

fi

121

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
(Blue line) For message sizes under 128 KiB,
the latency of BF2-to-BF2 communication ishigher than host-to-host communication
using conventional In niband NICs.
(Orange line) For messages under 256 KiB, it
is even slower to exchange messages
between the host and the BF2 on the same
node!
Similar ndings hold for multi-paircommunication and all-gather operations.
Thus, our best bet for getting any
performance improvement will be via offpath execution.

PA R T 3 : M I N I M D C A S E S T U DY

fi

fi

122

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
Even BF2 one-sided communication isslower than conventional In niband.

PA R T 3 : M I N I M D C A S E S T U DY

fi

123

SMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
The time to complete a BlueField-toBlueField all-gather is always worse—up to
3x—than via conventional NICs.

124

PA R T 3 : M I N I M D C A S E S T U DYSMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
The time to complete a BlueField-toBlueField all-reduce is always worse—up to
more than 4x—than via conventional NICs.

125

PA R T 3 : M I N I M D C A S E S T U DYSMARTER ALGORITHMS FOR SMARTER NETWORKS?

OSU benchmarks
imply off-path mode
For message sizes at 16 KiB or smaller, BF-toBF communication is slower thanconventional In niband.

126

fi

PA R T 3 : M I N I M D C A S E S T U DYSMARTER ALGORITHMS FOR SMARTER NETWORKS?

MiniMD “as-is” does
not benefit from BF2Each BF2 is a “mini-host.” Therefore,
consider an experiment in which we run
MiniMD using only the BF2 cards (i.e., nonode-host processing).
Because of its slower cores and worse
communication properties, MiniMD isalways slower than running it without
BF2, at all problem sizes (x-axis), any reneighboring interval (lines), and any node
con guration (subplot).
This type of result is characteristic of the
PENNANT study of Williams et al. mentioned
previously: without any algorithmic or code
restructuring, we should not expect any
bene ts.

fi

fi

127

PA R T 3 : M I N I M D C A S E S T U DY

Restructured method is a
viable simulation heuristic
Temperature divergence rate (TDR): a proxy
metric to assess the accuracy of our algorithm

128

Restructured method is a
viable simulation heuristic
Temperature divergence rate (TDR): a proxy
metric to assess the accuracy of our algorithm
We also veri ed that the computed results of
the restructured method are still within an
acceptable level of accuracy.

fi129