University of Tsukuba | Center for Computational Sciences

Software Researches for Big Data and Extreme-Scale Computing
System Software for Post-Peta Scale Computing / Extreme Big Data
Research and Development of this section is supported by: Development of System Software Technologies for post-Peta Scale High Performance Computing (www.hpcs.cs.tsukuba.ac.jp/project/crest-ppfs/en)
 EBD: Extreme Big Data – Convergence of Big Data and HPC for Yottabyte Processing (www.extreme-bigdata.jp)
 Statistical Computational Cosmology with Big Astronomical Imaging Data
Japan Science and Technology Agency

P-WAL: Parallel Write Ahead Logging

Efficient Summation on Encrypted DBMS

P-WAL is a new WAL protocol
designed for flash storage,
especially ioDrive.
P-WAL can cope with writeahead logging overheads suchas serialization and storage I/O.
P-WAL’s features are as follows:• Having no shared log buffers• Multiplexing storage I/O(SOSP 2015 Poster)• Existing encrypted
DBMS is inferior inperformance due toencryption.

Worker threads
wwwLogbuffers

LL
LioDriveLL

L
L L

L L

sum1

sumM

sum1

merge

sumM

sum1

• New interface - Sparse address space, atomic
batch operations and persistent trim• Simple design by fixed-size Region enabled bysparse address space and persistent trim
• Optimization techniques for object creation:
Bulk reservation and bulk initialization(GPC 2015)
Super Region Region
Region…
…

The network-based based parity
generation method utilizes
programmable abilities ofnetwork switches to minimize
overheads.
(1) Network-based Data
Processing Architecture for
Reliable and High-performance
Distributed Storage System(BigData Cloud 2015)

2

N

Next Region ID[]

NVM-BPTree: Highly Concurrent KVS
NVM-BPTree Highly concurrent Key-Value-Storerunning natively over Non-Volatile-Memory,
supporting range-queries.
KVS on NVM supporting rangequeries• Keys and metadata are storedIn-memory
in a in-memory B+Tree with
B+Treenegligible overhead.
• Support lock-free concurrentOperations. Search queries are
OpenNVM like Key-value store
not delayed by dynamicInterface
rebalancing of the tree.
NVM (Fusion-io flash device)

STEP 1: Put events to AISSTEP 2: Search from C with pruning
STEP 3: Memo events during search
STEP 4: Output events on leaf nodeThis work accelerates stream dataprocessing. It prunes unnecessary pathsin the active instance stacks, stores the A
information in the concise data structureformat, and finally expands in parallel forhigh efficiency.

a17a16a8a7a2

Bb18b15b13b11b9b6b5b3b1b0

Cyclo-Join is a distributed jointechnique that circulates one table S.
To speed up spatial join, Spatial
R S
Index are commonly used. We mix
R S
these join techniques for faster index R Sconstruction & similarity search.
SpatialApplications
R S
R S
IndexofRk・ Star catalogue cross-matching・ Guerrilla rainstorm prediction
R SR S6With 10 2D-points, our experiment
R S
shows x26.5 performance than thenaïve approach. (30 machines)

Writer187

Network82

Data-Aware Task Dispatching
Designed for Batch Queuing System
where its underlying file systems’
performance is strongly affected by thefile allocation.
NodeChosen
𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 = 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 × 𝛽𝛽
based onScore+ 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 × 1 − 𝛽𝛽
 𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇𝒇: calculated based on filenode
size and placement. load is the CPU
average.
storage(GPC 2015, IEEE Systems Journal 2015)
Parity #064

Pwrake: Data-intensive Workflow System

Pwrake workflow engine depends
on Gfarm file system for scalable
I/O performance. Task Schedulingschemes:(1) Locality-aware Scheduling using
Task
Multi-Constraint Graph
NetworkPartitioning (CCGrid 2012)
Ideal but (2) Disk Cache-aware Scheduling
Busyusing LIFO-based task queue
nodenode(Cluster 2014).
storagestorage

Workflow DAG Partitioning

XcalableACC (XACC) is a PGAS Language for accelerated parallel computers (e.g. GPU clusters), which is a directive-based language
extension of C and Fortran High productivity of programming by directives High performance by direct communication between ACCs

XACC Extensions

XcalableMP for distributed-memory parallelism

Data and work mapping onto multiple ACCs (two-leveldistribution)

XcalableMP (XMP) is a directive-based language extension
of C and Fortran for distributed-memory parallel systems• Distribution among nodes by XMP directives• Distribution among ACCs by the novel clauses. (on_device, layout, etc.)

OpenACC for offloading works for ACC

Direct communication between ACCs

OpenACC is another directive-based language extension forheterogeneous CPU/ACC systems• Communication directives from XMP accept data on the ACC memory.

Data and Work Mapping

HIMENO Benchmark with XACC

Array/Work

static float p[MIMAX][MJMAX][MKMAX], …;#pragma xmp template t(0:MKMAX-1, 0:MJMAX-1, 0:MIMAX-1)#pragma xmp nodes n(NDZ, NDY, NDX)
#pragma xmp distribute t(block, block, block) onto n#pragma xmp align p[i][j][k] with t(k, j, i)#pragma xmp shadow p[1:2][1:2][0:1]:#pragma acc data create(p, …){:#pragma xmp loop (k,j,i) on t(k,j,i)#pragma acc parallel loop reduction(+:gosa) collapse(2)for(i=1 ; i<imax-1 ; ++i)
for(j=1 ; j<jmax-1 ; ++j)
#pragma acc loop reduction(+:gosa) private(s0, ss)for(k=1 ; k<kmax-1 ; ++k){s0 = p[i+1][j][k] * …;
ss = … - p[i][j][k] …;
gosa += ss*ss;
wrk2[i][j][k] = p[i][j][k] + omega * ss;}:#pragma xmp reflect (p) acc:}

Definition of data mapping

Distribution among nodes#0CPUACC

Data allocation on ACC#1Comm.
between CPUs
Distribution among ACCs.

Direct Comm.
between ACCs

Loop parallelization andoffloading to ACC

Halo comm. on ACC

Performance of HIMENO Benchmark on HA-PACS/TCA
1600

GFLOPS (greater is better)

XACC extensions

45

XcalableACC

Basic Concept

235

Stripe #1

176

Stripe #0

C

c14
c12
c10
c4

Similarity Search Exploiting
Cyclo-Join + Spatial Index

sum

Network-based Parity Generation

1

QUERY
PATTERN SEQ(A a, B b, C c)
FROM sequence
RETURN abc

sumM

Object Storage Design in OpenNVMRegion

Input: b0 b1 a2 b3 c4 b5 b6 a7 a8 b9 c10 b11 c12 b13 c14 b15 a16 a17 b18 b19• We research highly
efficient query
processing on
encrypted DBMS.
split
Original data– We propose parallel
N parallelsummation scheme
thread1
thread2
threadN…
Bucket 1
Bucket 2
Bucket N
that fits encryptedgrouping
DBMS. It is 5.86 times
Task parallelsummation
faster than existing
grp1
grpM
grp1 grpM
grp1 grpM…
…
…
schemes.

LL

Fast Complex Event Processing over Streams

1400

MPI+OpenACC

1200

XACC

1000
800
600
400
200

CPU: Xeon-E5 2680v2 x 2
GPU: Tesla K20Interconnect:
Mellanox Connect-X3 Dual-port
QDRCompiler:
GCC 4.4.7, CUDA 6.5,
Omni XcalableACC compiler
MPI:
MVAPICH2-GDR 2.1a01x1x1 2x1x1 4x1x1 4x2x1 4x4x1 8x4x1 8x8x1

Division ( i x j x k )The performance of the XACC
version against MPI+OpenACC
version is more than 97%.

http://www.ccs.tsukuba.ac.jp/

3