Toward a Memory-Centric, Stacked Memory
Architecture for Extreme Scale, Data Intensive
ComputingJohn D. Leidel, Xi Wang, Yong Chen
February 4, 2017
Data-Intensive Scalable Computing Laboratory (DISCL)

Workshop on Pioneering Processor Paradigms

1

Overview• Background• Memory-Centric Architecture• Programming Models• Risks, Progress and Future Work2

Memory-centric Architecture Research

BACKGROUND

3

History of our HMC Efforts•

Our HMC-related research beganwith the GoblinCore-64 (GC64)
project
•

•
•

•

•

gc64.org

Purpose-built data intensive highperformance computing
instrument
Initial survey of high performance
memory technologies pointed
toward HMC as an excellent
candidate
GC64 local and global(partitioned) addressing relies
upon HMC physical memory
specification
Interconnect is a based upon an
extended HMC protocol & phy

.

.

.
.
GC64
Unit
. . . Task.
GC64
Thread.
. Task.
. TaskUnit
GC64
ThreadUnit.
.
Control.
.
GC64
Thread.
. Task Unit
Control
GC64
Unit
Thread
Task
Unit
.
.
Control
Unit Thread
Control
Unit
Control
Unit
Unit

.

......

MMU (Request Coalescing
Point)

.

.
.
GC64
Unit
. . . Task.
GC64
Thread.
. Task.
. TaskUnit
GC64
ThreadUnit.
.
Control.
.
GC64
Thread.
. Task Unit
Control
GC64
Unit
Thread
Task
Unit
.
.
Control
Unit Thread
Control
Unit
Control
Unit
Unit
MMU (Request Coalescing
Point)
Network on Chip

Off-Chip Network
Interface
HMC Channel
Interface

HMC Device

Data Intensive Scalable Computing
Laboratory at Texas TechAMO Unit

Software Managed Scratchpad

HMC Channel
Interface............

HMC Channel
Interface

HMC Device

4HMC Simulation Effort•

Significant lack of simulation
capabilities for HMC (andother emerging memory specs)•

GC64 is open source and BSD
licensed••

Prevented us from seeking Microninternal simulation capabilities

HMC-Sim was born!
••••

Functional simulation package
designed to implement the HMC
specification
Does not consider any one device
SKU
Packaged as a library written in C
Currently integrated into Sandia
SST toolkit••

John D Leidel and Yong Chen. HMC-Sim: A simulation
framework for hybrid memory cube devices. Parallel
Processing Letters, 24(04):1442002, 2014.
John D Leidel and Yong Chen. Toward memory centricarchitecture simulation for data intensive applications, 2015.
2015 Workshop on Modeling & Simulation of Systems andApplications.

5

HMC-Sim 2.0: CMC Operations•

Following several requests fromusers, we integrated the ability todefine Custom Memory Cube
operations in HMC-Sim•CMC Operations are packaged asshared libraries and loaded by the
HMC-Sim infrastructure at
runtimeReceive HMC
Packet Request

4 Link Average Barrier Latency
4 Link Maximum Barrier Latency
8 Link Average Barrier Latency
8 Link Maximum Barrier Latency

Handle HMC
Memory Operation

Is Tag-Bit
Operation?

No

40

Yes

Operations are handled just as any
traditional HMC packetized
operation using the definedcustom logic
Permits us to rapidly experimentwith future logic operations for
HMC (or potentially othermemory specs) • John D. Leidel and Yong Chen. HMC-Sim-2.0: A simulation platform

Decode Address

30

Build FE Bit
Request

Exit

Cycles

•

Start

50

Insert FE Bit
RequestNo

Retrieve Data
Payload

•

0

10

20

30

40
50
60
Number of Threads

70

80

for exploring custom memory cube operations. In 2016 IEEE
International Parallel and Dis- tributed Processing Symposium
Workshops (IPDPSW), pages 621–630, May 2016.
JohnLeidelandYongChen.Exploringtagbitmemoryoperationsinhybridmemory cubes. In Proceedings of the2016 International Symposium on Memory Systems, MEMSYS ’16,
New York, NY, USA, 2016. ACM.

Arbitrate Operation

No

Send Failed
Response

90

Success?

Yes

Update Memory
State

100

No

Update FE Bit StateIs Write
Rqst?
Yes

Operation
Complete
Update Data State

ReadEFROOT
TH 3

ReadEF

ReadEFIncFF
PARENT
TH 1

IncFF
WriteXE
WriteXF•

Insert Data Read
RequestRetrieve FE Bit

10

0

Yes

Is Read
Rqst?

20

PARENT
TH 4

IncFF

IncFF
LEAF
TH 0

LEAF
TH 2

LEAF
TH 5

LEAF
TH 6

ReadXX
SENSE

6

HMC to PIM (PnM)•

Following several efforts toexperiment with CMC
operations, we began totheorize what else waspossible
•
•
•

•

•

Can we utilize our CMC simulation
capabilities and our RISC-V
experience to build a PIM (PnM)?
What are the performance, power
and programmability ramificationsof doing so!?

Y

Z

HMC Link 0
HMC Link 1
HMC Link 2
HMC Link 3
System Link 0

Periphery IO
Devices

System Link 1
System Link 2
GC64 Socket

Communication primitives?
Additional AMO’s?
Concurrency features?

Our ongoing work in the
RISC-V community alongside
our CMC simulation efforts
triggered an interesting
thought•

X

GC64 Socket

GC64 Socket

System Link 3
HMC Link 0

Local Link 0

HMC Link 1

Local Link 1

HMC Link 2

Local Link 2

HMC Link 3

Local Link 3

System Link 0

Global Link 0

System Link 1

Global Link 1

System Link 2

Global Link 2

System Link 3

Periph. Link 0

HMC Link 0

Local Link 0

HMC Link 1

Local Link 1

HMC Link 2

Local Link 2

HMC Link 3

Local Link 3

System Link 0

Global Link 0

System Link 1

Global Link 1

System Link 2

Global Link 2

System Link 3

Periph. Link 0

Interconnect Router

Interconnect Router

HMC Link 0
HMC Link 1

Periphery IO
Devices

HMC Link 2
HMC Link 3
System Link 0
System Link 1
System Link 2
GC64 Socket

System Link 3
X

YZJohn D. Leidel, GoblinCore-64: A Scalable, Open Architecture for Data
Intensive High Performance Computing. Doctoral Dissertation. May
2017.

7

Merging Stacked Memories and Lightweight RISC Cores

MEMORY-CENTRICARCHITECTURE
8

Memory-Centric Architecture
Thought Experiment•

Construct a memory-centric, processor near memory architecture using HMC
and open source RISC-V cores•••

Construct the device using reasonable extensions to the existing Gen2 HMC
specification
•
•
•

•

RISC-V devices have excellent power/performance efficiency and we have existing hardware
implementations that are BSD licensed and proven in silicon
The HMC device specification isn’t perfect, but it has been proven in silicon using Micron’s
production-scale fab

The Gen2 spec has a reasonable number of unused opcodes that can be used to extend the existing
spec without drastically expanding the scope of the logic layer
Avoid performing protocol translations when communicating off a local cube (eg, AXI)
Minimize the disparate hardware modules

Demonstrate the approach using traditional and novel programming models
using known simulation methods•••

Using the existing HMC-Sim and RISC-V Spike simulators, demonstrate a reasonable simulationof our environment
Utilize existing PGAS and shared memory approaches (UPC, Chapel, OpenMP)
Utilize novel dataflow programming models

9Memory Architecture•

Consists of 8-Link, 8GB HMC
devices constructed in two
configurations
Processing Nodes (1a)

•
•
•

•

Implements processing capabilities in the
logic layer of an HMC device
HMC DRAM storage utilized for application
workload memory/working set

Router Nodes (1b)•••8GB HMC Memory Storage

Implements a modified HMC packet
specification to include our extended memory
request operations
HMC DRAM storage utilized for routing
tables, configuration information, etc

HMC
Vault
Logic

RISC-V
Processing
Array
Figure 1a

HMC
GPIO

8GB HMC Memory Storage
Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

HMC
GPIO

HMC
Vault
Logic

HMC Router
Logic

Figure 1b

Logic area is doubled beyond thebase HMC device implementation•

34 mm2 to 68 mm2•

Additional die area utilized to implement
multicore RISC-V SoC on processing
nodes
Additional die area utilized to implement
routing logic on router nodes•

https://www.extremetech.com/computing/167368-hybrid-memory-cube-160gbsec-ram-startsshipping-is-this-the-technology-that-finally-kills-ddr-ram

10Processing Architecture•

Processing nodes feature [28]RISC-V cores in the logic layer
Cores are configured using the RISC-V
IMAFD (G) spec
Currently utilize the Rocket 5-stage, inorder pipeline

•
•
•

•

•

•
•

L1I$

Simple, but effective!

34GFlops each at ~1Ghz and 1.2mm2(28nm)

Configured with SIMD
extensions to increase processing
throughput••

HWACHA SIMD
Unit

HWACHA SIMD unit
Could be substituted for theforthcoming RISC-V SIMD/vector spec

Peak performance per core:
34GFlops
Peak performance per 3D stack:952GFlops

RISC-V CoreHMC XBAR Interface•

Each core contains a local L1
instruction cache, but no data
cache

•

Why?
•

•

Logic area is incredibly precious inour device; we don’t want to
expend area on local cachecoherency
We don’t want to expend
communication bandwidth tomaintain cache coherency
11

Interconnect Architecture•Interconnect is based upon
HMC methodology adapted
for long-haul connectivity
Constructed using [4] processing
nodes and [2] routing nodes in a 3D
torus

•

•

X Y Z

HMC Vault
Logic

RISC-V
Processing Array

Connectivity is split into four
sets of links
2 links per device connected torouters
6 links per device connected toadjacent 3 nodes

•
•
•

•
•

Local connectivity promotes tiered
NUMA

Each router has 3 links for [X,Y,Z]
torus connectivity to adjacentrouters
One link per router dedicated toperipheral I/O connectivity

HMC
GPIO

HMC Vault
Logic

HMC
GPIO

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

HMC
GPIO

HMC Vault
Logic

HMC Router
Logic

HMC
GPIO

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

HMC
GPIO

HMC Vault
Logic

HMC Router
Logic

HMC
GPIO

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

RISC-V
Processing Array

HMC Vault
Logic

RISC-V
Processing Array

Periphery I/O

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

RISC-V
Processing Array

HMC Vault
Logic

Compute to Router
Router to Router
Router to I/OCompute to Compute

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

Periphery I/O

X Y Z

12

Interconnect Packet Structure
Texas Tech University, John D. Leidel, May 2017

Texas Tech University, John D. Leidel, May
Extended Header(8-bytes)

Extended Tail
(8-bytes)

HMC Packet

Table 3.2: GoblinCore-64 System Inteconnect Packets

Figure 3.10: GC64 Extended HMC Packet

Full complement of partitioned global
Command
Opcode Request FLITS Response FLITS
Write
0x4func3
2
hasthebeen described
works in order16-byte
to extend
the in-situ atomicity
address packets tructureusing
GC64in previous
extended
32-byte
Write
0x5
4
2
tions [99] [92], for graph traversals [118] and for processing near memory [150] [119] [70] [94].
physicaladdressing
spec
48-byte Write
0x6
5
2
Table 3.3: GoblinCore-64 Atomic System Inteconnect Packets
In order to support our extended memory addressing and messaging capability, we utilize

64-byte Write

0x7

96-byte Read
0x3B
112-byte
Read
0x3C weJohn D.AsLeidel,
Xi of
Wang,
and Yong Chen.
a result
the aforementioned
system interconnect
designand implementation,
128-byte
Read
0x3D
GoblinCore-64:
Architectural
Specification.
may summarize the performance of our proposed GoblinCore-64 platform in terms of theTechnical report, Texas Tech University,
256-byte Read
0x3E
3.2.4

•

Texas Tech University, John D. Leidel, May 2017

Command
OpcodeFLITS forResponse
FLITS
the in-situ
packet Request
specification
memoryreads, writes, posted writes and atomic opera80-byte Write
0x14
Dual 8-byte Signed Add Imm
0x70
3
2
Gen2 HMC3 specification [17].
For each96-byteof the aforementioned
packettypes,
Single 16-byte Signed Add Imm tions in the
0x71
2
Write
0x15
Posted dual 8-byte Signed Add Immthe HMC 0x72
30
specification encodes a non-zero number of112-byte
flow control
units, or FLITS,
for theWrite
0x16
Posted single 16-byte Signed Add Imm
0x73
30128-byte
Write
0x17
request packet
some number of non-zero FLITs for the reDual 8-byte Signed Add Imm And Return
0x74 and (for 3non-posted requests)
3
256-byte
0x20
Single 16-byte Signed Add Imm andsponse
Return packet.
0x75 Each FLIT
3 is designated3as 16-bytes ofdata thatWritecontains all the necessary
8-byte Increment
0x76
2
2
16-bytePostedWrite
0x24 As
locality, opcode
information for the respective request.
Posted 8-byte Increment
0x77 (request
2 type) and addressing
032-byte Posted Write
0x25
16-byte XOR
0x55 3.10, we
3 extend this functionality
3
shown in Figure
to include an additional FLIT for each
48-byte Posted Write
0x26
16-byte OR
0x56
33
request andresponse packet,
respectively,
the standard message opcodes. We
16-byte NOR
0x57
3
3 for each of64-byte
Posted Write
0x27
16-byte AND
3 space present3in the HMC specification for these custom memutilize the0x58
unused opcode
80-byte Posted Write
0x29
16-byte NAND
0x59
3
3
ory
cube,
or
CMC,
operations.
The
extended
payloadspace
isutilized
toencodethesource96-bytePostedWrite
0x2A
8-byte Compare & Swap (GT)0x5A
3
3
16-byte Compare & Swap (GT)0x5B
3
3112-byte
PostedWrite
0x2B(requester) routing information in the packet tail and the destination (endpoint) routing in8-byte Compare & Swap (LT)0x5C
3
3128-byte
formation0x5D
in the packet 3header. As a result,each GC64
socketPosted
has theWriteability to 0x2C
dispatch
16-byte Compare & Swap (LT)3256-byte
PostedWrite
0x2D
8-byte Compare & Swap (EQ)
0x5Eextended memory
3
rudimentary,requests3using our extended HMC request structure to any
16-byte Compare & Swap (ZERO)
0x66
3
3
16-byte Read
0x2E
adjacent nodeinterconnect. We summarize the opcode
8-byte Equal
0x67 connected
3 to the global memory
232-byte Read
0x2F
16-byte Equal
3 FLIT length for
2 each of our extended memory operations in Taencodings0x6B
and associated48-byteRead
0x38
8-byte bit write
0x6C
3
2
ble
3.2
and
Table
3.3.
64-byte Read
0x39
Posted 8-byte Bit Write
0x6D
3
08-byte Bit Write with Return0x6E
3
380-byte Read
0x3A
16-byte Swap
0x6F
3
3

System Interconnect Scalability

memory2015.
capacity, local memory bandwidth, global interconnect bandwidth and the bisecSeptember
tion bandwidth. We present our results of this analysis in Table 3.4.

6
7
8
9
10
18
3
4
5
6
7
8
9
10
18
2
2
2
2
2
2
2
2
2

2
2
2
2
2
2
0
0
0
0
0
0
0
0
0
3
4
5
6
7
8
9
10
18

13

RISC-V
Processing Array

Performance Potential

4
Link
5
Link
6
Link
7

X Y Z

Figure 3. Stacked Memory Node Architecture
TABLE 1. S YSTEM A RCHITECTURE•

Example peak performancecaveats:••••

Maximum local scalar bandwidth islimited to 50% of peak DRAM
bandwidth (16-byte HMC fetch)
Maximum local bandwidth can beachieved using HWACHA SIMD
ops
15Gbps SERDES links de-rated toaccount for DRAM/logic latency(40GB/s per link)

Device power is based uponmeasured HMC values andmeasured RISC-V values••

Does not account for process
differences or process
improvements
Does not account for long-haul
SERDES connectivity, eg,translation to optical

Processing Architecture
Component
Value
RISC-V Rocket with HWACHA Area
1.2mm2
RISC-V Rocket with HWACHA Freq
951Mhz
RISC-V Rocket with HWACHA Perf
34 GFlops
RISC-V Cores per HMC
28
Peak RISC-V Performance
952 GFlops
Memory Architecture
Component
Value
HMC Configuration
8Link-8GB Device (15Gbps)
HMC Area (Compute and Router)
68mm2
System Architecture
Configuration
Value
HMC Power
136watt
5x5x5 Torus Performance
476TFlops
5x5x5 Torus Memory
1TB
5x5x5 Torus Power
102KW
10x10x10 Torus Performance
3.8PFlop
10x10x10 Torus Memory
8TB
10x10x10 Torus Power
816KW
50x50x50 Torus Performance
476PFlop
50x50x50 Torus Memory
1PB
50x50x50 TABLE
Torus Power
103MW
2. S YSTEM A RCHITECTURE P ERFORMANCE
System Architecture
Configuration
Total Network Bandwidth
5x5x5 Torus
30 TB/sdevices such as infrastructure
network(Ethernet) devices,
10x10x10 Torus
240 TB/smass storage devices
or application-specificsensor devices
50x50x50
Torus
30 PB/sdirectly into the memoryand compute
Configuration
Bisection hierarchy.
Bandwidth
5x5x5 Torus
4 TB/sTable I summarizes
the system architecturaldetails as
10x10x10 Torus
16 TB/s
50x50x50notable
Torus
4 PB/swell as provides somescaling
parameters for 53 ,Peak Bytes/Flop
103 and 503 cubic,Configuration
3D torus topologies.
While our largest
5x5x5 Torus
0.063
10x10x10 Torus
0.063
50x50x50 Torus
0.063

cache linrequests.
request is
write reqnear-mem
pay the clinks forstill incu
for scalaelement i
load/storeof the att
nipulate 1
and store
are wellenforcedremote m
are yet agof contro
characterpotential
In ad
can also ccharactersive appl
interconn
bandwidt
highest p
generatioThisinability
productionmup to 40
efforts
barring
note ourIn additio
Each systeof tproposed
withasi
overall system
In thism
as this,the tramemoryprote
betweenorthogonal
Alsogito the core croutedthscale
prototypintegrate a cebilities in14
orde
memory manasuch, we musMapping Communication and Dataflow Paradigms

PROGRAMMING MODELS

15

Programming Models for PIM•

Many different programming
models utilized across historical
PIM (PnM) architectures
•
•

•

•

We have become more adept at
expressing parallelism inprogramming models
Data flow, SIMD compilation,
SIMD+MIMD (OpenMP 4.X+),
Cilk, MapReduce, etc.

•

Not all were created equal
Not all were successful•

Thinking Machines: Lisp
ICL/CPP DAP: Vectorized Fortran;
templated C++
Terasys: dbC (C-extensions)•

What didn’t?

EMU Gossamer: Cilk-style threadlets
ISI DIVA: distributed shared memory
model

•

Berkeley CRAM/IRAM••PGAS languages are a great step
forward, but they are not widely
accepted

Virtual memory is a MAJOR issue

•

SIMD:
•

We are not yet good at expressing
locality

•

MIMD:
•
•

•

What worked?

Bit serial:
•
•

•

•

Is just dealing with physicaladdressing the solution?

Communication (NUMA) latencyis hard to express in the instruction
set

16

Compute-Communication/SPMD
Programming Model•

We want to experiment with porting
very traditional languages to thearchitecture
Shared Memory•

OpenMP 5.x proposed spec has additionalclauses to specify certain degrees of physicalmemory locality••

Specified on a per-variable basis using clause
modifiers

We would like to experiment with adapting
these for PIM architectures
OpenMP has been utilized in graph algorithm
research (see GAP benchmark suite)•••

Partitioned Global Address Space
Given our adoption of partitioned physicaladdresses (from GC64), PGAS languages area natural fit
UPC is the first PGAS language target

•
•
•

•
•
•

#pragma omp memkind(fastmem : val: a ,b, persistent: ref: c,d)#define N <something large>
shared int v1[N], v2[N], v3[N]
upcforall( i=0; i<N; i++ ){v3[i] = v1[i] + v2[i];}Berkeley UPC translator and runtime

Chapel may follow
Our biggest concern: minimizing the overhead
from the PGAS runtime
UPC is not traditionally known for graphalgorithm/data intensive computing research
http://www.openmp.org
http://upc.lbl.gov/

17

Data Flow Programming Model•

Data flow model similar to the Emu
execution model•

Move the compute to the memory•

Cores are permitted to execute providing they
access memory in local cube storage•

Remote memory accesses trigger a fault
•
•
•
•

•

Faults trigger the equivalent to an RPC request to aremote core co-located with the remote memory
Local core builds a message block including the PC
and the remote memory location
Message is sent to the remote device and queued
Remote core picks up the request and begin execution

Implications?
•••

HMC Memory Storage

HMC Vault
Logic

RISC-V
Processing Array

Application is copied everywhere in order to avoid
moving the entire binary.
Applications must efficiently utilize locality whenpossible
Graph traversals/pointer chasing

HMC Link
Interconnection
Network

HMC Memory Storage

HMC Vault
Logic

Target apps:
•

HMC
GPIO

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

Memory fault request
encapsulated as RPC
request and bundled
as HMC packet

RISC-V
Processing Array

HMC
GPIO

Link
0
Link
1
Link
2
Link
3
Link
4
Link
5
Link
6
Link
7

Fault request is
received and local
processor core jumps
to the desired target
routine18

Future Research Directions

RISKS, PROGRESS AND
FUTURE WORK
19

RisksSoftware•Near-memory architectures arenotoriously difficult to program••

Hardware••

What additional programming models orfeature support do we need to promotelocality?

•

System software?
•••

Memory protection, virtualization,
allocation are all difficult tasks when
operating close to memory
Supervisor versus user-mode execution?

Tools?
••

Debugging PIM/PnM architectures canbe especially tricky
How do we exploit temporal and spatial
memory locality in our compiler
optimizations?

Hardware IP licensing••

TSV technology is highly processspecific and expensive to license
RISC-V, GC64 and our otherdesigns are BSD licensed
TSV providers may not be amenable
to an open source hardware design

HMC DRAM Layer Licensing••

Analog DRAM logic andmanufacturing IP is highly
proprietary
Difficult to persuade few remaining
DRAM manufacturers to license
only the DRAM portion of thedevice20

Current Progress/Future Work•

Initial prototyping efforts
have begun to marry the
HMC-Sim infrastructure
and the RISC-V Spikesimulator••

Fairly significant effort to make
HMC-Sim centric in thesimulation infrastructure (asopposed to Spike

Additional prototyping isunder way to produce a set
communication packet
specification for HMC•

Forthcoming publications (lateSpring/early Fall)

DRAM
Logic

HMC-Sim

CMC Interface

Spike

CMC Interface

Spike

Vault
XBAR
CMC Interface
Logic Interface
.
.
.

Spike

CMC Interface

Communication
Logic

Figure 5a: Compute Node

DRAM
Logic

HMC-Sim

CMC Interface

Communication
Logic

CMC Interface

Communication
Logic

Vault
XBAR
CMC Interface
Logic Interface.
.
.
CMC Interface

Communication
Logic

Communication
Logic

Figure 5b: Router Node21

Questions/Comments?

John Leidel
john.leidel<at>ttu.edu
http://gc64.org
http://discl.cs.ttu.edu/gitlab/groups/gc64

22