博士論文
Doctoral Dissertation

Task Parallelism at Scale:
Locality-Aware Scheduling and Global Address Space（局所性に配慮したスケジューリングと大域アドレス空間による大規模タスク並列処理）

椎名 峻平
Shumpei Shiina
学籍番号48-217405

東京大学大学院 情報理工学系研究科 電子情報学専攻

Department of Information and Communication Engineering
Graduate School of Information Science and Technology
The University of Tokyo

指導教員

田浦 健次朗 教授

AdvisorProf. Kenjiro TauraAbstract
Computer programming has evolved to abstract away low-level hardware details, focusing instead on algorithms
and logic. However, even today, low-level programming is still required for high-performance computing on
massively parallel computers, including supercomputers. In particular, programmers are generally required
to explicitly distinguish between multi-core parallelism on shared memory and cluster-level parallelism on
distributed memory. This requires developing two separate parallelization schemes for a single application.
This situation arises from the challenges in designing programming models and their runtime implementations
that are highly productive while simultaneously achieving high performance.
One promising approach to this issue is task-parallel programming, in which parallel computations are
expressed in terms of logical series-parallel relationships between tasks. Tasks are automatically load-balanced
across physical processors by the underlying runtime system, allowing programmers to concentrate only on
logical parallelism. Until now, many eﬃcient task-parallel runtime systems have been developed, showing
great scalability to many-core CPUs with tens or hundreds of cores. Nevertheless, most task-parallel runtime
systems have been designed for shared-memory computers, not for distributed-memory computers. Therefore,
their scalability has been eﬀectively limited to a single machine.
This dissertation presents how to eﬃciently scale task parallelism beyond machine boundaries. We demonstrate this by developing Itoyori, a distributed task-parallel runtime system. While many existing runtime
systems share a similar goal, Itoyori ambitiously seeks to achieve both productivity and performance at ahigh level. The high productivity of Itoyori is derived from its simple and intuitive programming model,
which integrates global task parallelism—in which tasks are automatically load-balanced across machineboundaries—and global address space—in which physically distributed data can be uniformly addressed andaccessed from any machines. However, naively combining these two concepts easily results in poor performance due to bad data locality. This dissertation discusses two data locality challenges and runtime-based
approaches to them.
The ﬁrst locality challenge lies in sharing data among tasks. A key to achieving high performance on
distributed memory is to reduce the amount of communication and increase the granularity of communication.
However, if tasks can be moved to diﬀerent machines for dynamic load balancing, it becomes diﬃcult
for programmers to aggregate communication for the same data accessed by multiple tasks, even if they
are eventually scheduled on the same machine. Consequently, each task issues global memory accesses
individually for its own data, resulting in redundant and ﬁne-grained communication. This is because today’s
typical global address space implementation directly translates global memory accesses into communication
requests to remote machines. Itoyori addresses this issue by providing a software cache for global memory
accesses, enabling eﬃcient cache sharing among parallel tasks running on the same processor.
The second locality challenge is the mismatch between task locations and data locations. Given that data arephysically distributed to multiple machines, the task scheduler should map tasks to the machines where theirdata reside. Moreover, today’s machines often feature Non-Uniform Memory Access (NUMA) architecturesand hierarchical caches. This requires scheduling tasks that share the same working set to close locations in
the memory hierarchy. We address these data locality requirements for deep memory hierarchy by a singleitask scheduler, called Almost Deterministic Work Stealing (ADWS). First, ADWS schedules tasks to processors
deterministically so that the task hierarchy matches the memory hierarchy. It then performs dynamic loadbalancing by stealing tasks from as close processors as possible.
With these two locality improvements combined, Itoyori has achieved high scalability, eﬃciency, andpractical usability on distributed memory. We demonstrate this through experiments with various benchmarks
and real-world applications. Our benchmark programs and applications have been straightforwardly ported
from shared-memory task-parallel implementations to Itoyori without major changes to their overall parallel
algorithms. This demonstrates great productivity of Itoyori, as it accommodates both shared memory anddistributed memory with single-level parallelized code. Our experiments showed that they successfully scaled
to multiple machines, particularly if parallel algorithms exhibit a high degree of locality. Notably, the N-bodysimulation experiment showed up to a 10× speedup for strong scaling from a single machine to 12 machines,
and maintained 75% parallel eﬃciency on 512 machines for weak scaling. These results demonstrate that
Itoyori can strike a good balance between productivity and performance for large-scale computing.

ii

Acknowledgments
I have been incredibly fortunate to have been supervised by Professor Kenjiro Taura for about six years,
starting from my undergraduate studies. I have learned a lot of skills and knowledge from him, ranging
from basic technical know-how in the ﬁelds of system software and high-performance computing, to how topresent research in an appealing manner. I particularly admire his ability to discern the essence of matters andarticulate sharp, insightful observations in concise and often humorous language. I am deeply grateful for him
for introducing me to the exciting research ﬁeld of task-parallel runtime systems and for his patient support
throughout my doctoral journey.
I am also profoundly grateful to the members of my doctoral thesis committee: Professor Shuichi Sakai,
Professor Toyotaro Suzumura, Professor Yoshihiro Kawahara, Professor Hidetsugu Irie, Professor Shigeru
Chiba, Professor Rio Yokota, Professor Toshihiro Hanawa, Lecturer Yusuke Matsui, and Professor Kenjiro
Taura. The feedback I received during the predefense was immensely insightful and constructive, whichstrengthened the presentation of my dissertation. In particular, I am thankful to Professor Toyotaro Suzumura
and Professor Hidetsugu Irie, who assisted me during the advisor meetings in the early stages of my doctoral
research. Their valuable comments and encouragement were a strong motivation for me to continue advancing
my research. Furthermore, it was extremely beneﬁcial to have detailed discussions with Professor Rio Yokota
about the application ExaFMM and with Lecturer Yusuke Matsui about ANNS, from their expert perspectives.
Thanks to these discussions, the quality of the evaluations in my dissertation was undoubtedly improved.
The members of Taura Laboratory were indispensable to my research life. Associate Professor Shigeyuki
Sato, with his expertise in programming languages, provided insightful and intriguing comments on the Itoyori
APIs and the positioning of my research, which greatly helped me in reassessing my work objectively. Hiscomments on my research presentation were also very reasonable and helpful. While I enjoyed lunch breaks
with many labmates, I frequently had lunch with Takato Hideshima, Ryota Kobayashi, and Godai Takashina,
discussing technical topics like programming and typesetting, which was really enjoyable. In particular, Takato
Hideshima generously provided me with detailed and useful tips on the C++ standard. The Itoyori C++ APIs
would not have been what it is today without him. Yukio Siraichi has been not only a labmate for many years
but also a treasured friend, with whom I have had various conversations about research and life. During mytime as a newbie from undergraduate to master’s studies, I owe a lot to Dr. Shintaro Iwasaki and Dr. Wataru
Endo, whose research areas closely aligned with mine and have signiﬁcantly inﬂuenced my current research. I
also express my appreciation to Sachie Ikeya, the secretary in the lab, for handling numerous budget proceduresand for encouraging me. While it is not possible to name everyone, I am truly grateful to all my other labmates
as well.
Finally, I would like to express my gratitude to my family for raising me and supporting me throughout thislong PhD journey. I wish the greatest happiness to everyone who has supported me thus far.

iii

Contents
1. Introduction

1

1.1. Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2. The Itoyori Runtime System and Its Challenges . . . . . . . . . . . . . . . . . . . . . . . . .
1.3. Overview and Contributions of This Dissertation . . . . . . . . . . . . . . . . . . . . . . . .
2. Background and Preliminaries

1
2
3
6

2.1. Modern Architectures of Multi-Core CPUs and Clusters . . . . . . . . . . . . . . . . . . . . .
2.1.1. Hierarchical Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2. Non-Uniform Memory Access (NUMA) Architectures . . . . . . . . . . . . . . . . .
2.1.3. High-Performance Interconnects and Remote Direct Memory Access (RDMA) . . . .
2.2. Task Parallelism and Work Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1. Fork-Join Task Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2. Work Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.3. User-Level Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.4. Data Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.5. Distributed Task-Parallel Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3. Global Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1. Distributed Shared Memory (DSM) . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2. Partitioned Global Address Space (PGAS) . . . . . . . . . . . . . . . . . . . . . . . .
3. Programming with the Itoyori Runtime System

3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2. Global Task Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1. Switching between SPMD and Task-Parallel Scopes . . . . . . . . . . . . . . . . . .
3.2.2. Fork/Join APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3. Global Address Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1. Memory Allocation and Distribution Policies . . . . . . . . . . . . . . . . . . . . . .
3.3.2. Checkout/Checkin APIs for Global Memory Access . . . . . . . . . . . . . . . . . . .
3.3.3. Release/Acquire APIs for Synchronization . . . . . . . . . . . . . . . . . . . . . . . .
3.4. High-Level C++ APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1. Checkout Spans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2. Global Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.3. Parallel Loops and Global Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5. Pitfalls and Current Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4. Scalable Continuation Stealing on Distributed Memory

6
6
9
10
11
12
15
16
17
18
19
19
20
22

22
22
23
24
24
25
26
27
27
29
29
30
33
34

4.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

iv

Contents
4.2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1. Child Stealing and Continuation Stealing . . . . . . . . . . . . . . . . . . . . . . . .
4.2.2. Handling Task Synchronization (Join) . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.3. Suspending and Migrating Threads . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.4. Uni-Address Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3. Joining Threads over RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1. Joining Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.2. Freeing Remote Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4. Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1. Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.2. Implementations of Child Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.3. Synthetic Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5. Performance Analysis and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.1. Performance Analysis of Joining Strategies . . . . . . . . . . . . . . . . . . . . . . .
4.5.2. Child Stealing vs. Continuation Stealing . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.3. Unbalanced Tree Search (UTS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.4. Longest Common Subsequence (LCS) . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5. Leveraging Locality of Global Memory Accesses via Software Caching

5.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2. Rationale of Checkout/Checkin APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3. Implementation of the PGAS Library and Software Cache . . . . . . . . . . . . . . . . . . .
5.3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.2. Memory Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.3. Global View Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.4. Cache Coherence Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4. Integration with Work Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1. Release/Acquire Fences in Work Stealing . . . . . . . . . . . . . . . . . . . . . . . .
5.4.2. Lazy Execution of Release Fences . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

6.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.1. Motivating Example: Decision Tree Construction . . . . . . . . . . . . . . . . . . . .
6.2.2. Notation and Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.3. Two Types of Data Locality in Fork-Join Parallelism . . . . . . . . . . . . . . . . . .
6.3. Almost Deterministic Work Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1. Deterministic Task Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.2. Dynamic Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.3. Issue of Shared Cache Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4. Multi-Level Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.1. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

35
36
36
37
37
38
38
41
42
42
42
43
45
45
45
47
49
52
53

53
54
55
55
56
56
59
59
59
60
61
63

63
65
65
66
67
67
67
71
73
74
74

Contents

6.5.
6.6.

6.7.
6.8.

6.4.2. Design and Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Multi-Level ADWS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Performance Evaluation on Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.1. Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.2. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.3. Overall Performance Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.4. Sensitivity to Imprecise Work Hints . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.5. NUMA Memory Policy for Deterministic Scheduling . . . . . . . . . . . . . . . . . .
Distributed-Memory Implementation in Itoyori . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7. Application Studies and Performance Evaluation

7.1.
7.2.
7.3.
7.4.
7.5.
7.6.
7.7.

75
76
78
78
79
81
84
86
87
88
89

Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Cilksort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Heat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
UTS-Mem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
ExaFMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
ANNS Graph Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
PageRank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

8. Related Work120

8.1. Distributed Work Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
8.2. Global Address Space and Software Caching . . . . . . . . . . . . . . . . . . . . . . . . . . 120
8.3. Locality-Aware Task Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
9. Conclusions

123

Appendix A. Itoyori Program Examples

125

A.1. Cilksort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Appendix B. Supplemental Experimental Results

128

B.1. ANNS Graph Construction with the msspacev-10M Dataset . . . . . . . . . . . . . . . . . . . 128
B.2. Proﬁling Result of PageRank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

vi

List of Figures
2.1. Memory hierarchies of modern CPU architectures. . . . . . . . . . . . . . . . . . . . . . . .
2.2. Local and remote memory access in Non-Uniform Memory Access (NUMA) architectures. .
2.3. Conventional socket-based communication over TCP/IP and Remote Direct Memory Access(RDMA) technology that bypasses the kernel. . . . . . . . . . . . . . . . . . . . . . . . . .
2.4. Typical task-parallel APIs in C++ libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5. Recursive parallel merge sort program in C++ with parallel invoke(). . . . . . . . . . . .
2.6. Example of task parallelism that is not fork-join. . . . . . . . . . . . . . . . . . . . . . . . .
2.7. Example of fork-join parallelism expressed in OpenCilk [10]. . . . . . . . . . . . . . . . . .
2.8. Directed acyclic graph (DAG) of fork-join parallelism. . . . . . . . . . . . . . . . . . . . . .
2.9. Task tree of fork-join parallelism at a certain point in time. . . . . . . . . . . . . . . . . . . .
2.10. Demonstration of task scheduling by work stealing (work-ﬁrst policy). . . . . . . . . . . . . .

10
12
13
14
14
14
14
15

3.1. Itoyori APIs for switching between the SPMD and fork-join scopes. . . . . . . . . . . . . . .
3.2. Fibonacci example with Itoyori’s low-level multithreading APIs. . . . . . . . . . . . . . . . .
3.3. Low-level memory allocation APIs in Itoyori. . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4. Recursive parallel merge sort implemented in Itoyori using low-level checkout/checkin APIs. .
3.5. Usage of low-level release/acquire APIs with fork/join APIs and checkout/checkin APIs. . . .
3.6. Recursive parallel merge sort implemented in Itoyori using the checkout span. . . . . . . . . .
3.7. Global vector and span containers in Itoyori. . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.8. High-level transform call in Itoyori (corresponding to a loop in Figure 3.7). . . . . . . . . . .
3.9. Itoyori’s for each() call and global iterators (equivalent to Figure 3.8). . . . . . . . . . . . .
3.10. AXPY and L2 norm calculation using high-level parallel loops and global iterators. . . . . . .

23
24
25
26
28
29
30
30
31
31

4.1.
4.2.
4.3.
4.4.
4.5.
4.6.

36
38
39
40
43

Comparison of child stealing and continuation stealing. . . . . . . . . . . . . . . . . . . . . .
Stack management in uni-address threads. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation of stalling join over RDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation of greedy join over RDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Pseudocode of PFor and RecPFor benchmarks. . . . . . . . . . . . . . . . . . . . . . . . . .
Performance comparison between diﬀerent joining strategies and steal strategies with synthetic
benchmarks (PFor and RecPFor). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7. Time series of scheduler activities in RecPFor benchmark (𝑁 = 222 ) on ITO-A with 576 cores(16 nodes). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.8. Throughput of UTS benchmark on ITO-A (up to 9,212 cores). . . . . . . . . . . . . . . . . .
4.9. Throughput of UTS with Itoyori on Wisteria-O. . . . . . . . . . . . . . . . . . . . . . . . .
4.10. Task dependency of LCS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.11. Algorithm for LCS based on recursive decomposition and futures. . . . . . . . . . . . . . . .
4.12. Execution times of LCS with continuation stealing (greedy join). . . . . . . . . . . . . . . . .

vii

6
9

44
47
48
50
50
50
51

List of Figures
5.1.
5.2.
5.3.
5.4.
5.5.

Cache management for diﬀerent PGAS APIs. . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of memory management in Itoyori. . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation of checkout/checkin operations. . . . . . . . . . . . . . . . . . . . . . . . . .
Possible release/acquire fences in fork-join calls. . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation of lazy execution for release fences. . . . . . . . . . . . . . . . . . . . . . . .

54
55
57
60
60

Algorithm for parallel decision tree construction. . . . . . . . . . . . . . . . . . . . . . . . .
Simpliﬁed model for fork-join parallelism. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example of a DAG for decision tree construction. . . . . . . . . . . . . . . . . . . . . . . . .
Overview of deterministic task mapping in (single-level) ADWS with four workers. . . . . . .
Speciﬁcation of work and space hints for (multi-level) ADWS. . . . . . . . . . . . . . . . . .
Classiﬁcation of tasks in a cross-worker task group. . . . . . . . . . . . . . . . . . . . . . . .
Algorithm for deterministic task mapping in ADWS. . . . . . . . . . . . . . . . . . . . . . .
Illustration of the task execution order for worker 𝑖 in ADWS, with the tasks numbered in thatorder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.9. Example of execution stalls in the deterministic task mapping of ADWS. . . . . . . . . . . . .
6.10. Distribution tree for dynamic load balancing in ADWS. . . . . . . . . . . . . . . . . . . . . .
6.11. Algorithm for getting a runnable task in ADWS. . . . . . . . . . . . . . . . . . . . . . . . . .
6.12. Example of a tree of caches with 16 cores. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.13. Algorithm for execution of a task group in multi-level scheduling. . . . . . . . . . . . . . . .
6.14. Illustration of multi-level ADWS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.15. Algorithm for cache hierarchy ﬂattening in multi-level ADWS. . . . . . . . . . . . . . . . . .
6.16. Speedup (or FLOPS) on 56 cores with various working set sizes. . . . . . . . . . . . . . . . .
6.17. Breakdown of the execution times proﬁled by the schedulers (corresponding to the rightmost
points in Figure 6.16). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.18. Cache miss counts proﬁled by perf system calls (corresponding to the rightmost points in
Figure 6.16). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.19. Sensitivity of imprecise work hints for various unbalanced computation graphs. . . . . . . . .
6.20. Evaluation of ADWS when no work hints are provided by programmers. . . . . . . . . . . . .
6.21. Performance comparison between the NUMA interleave policy and the local allocation policy
with ADWS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.22. Distributed-memory representation for a distribution tree in ADWS. . . . . . . . . . . . . . .

65
66
66
67
68
68
69

6.1.
6.2.
6.3.
6.4.
6.5.
6.6.
6.7.
6.8.

70
71
71
73
74
76
76
77
82
82
83
85
85
85
87

7.1. Execution time of Cilksort (1G elements) with various cutoﬀ counts and caching policies. . . . 91
7.2. Execution time of Cilksort with strong scaling. . . . . . . . . . . . . . . . . . . . . . . . . . 92
7.3. Parallel eﬃciency of Cilksort with strong scaling. . . . . . . . . . . . . . . . . . . . . . . . . 93
7.4. Proﬁling of Cilksort with strong scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.5. Execution time of Cilksort with weak scaling (1G elements per node). . . . . . . . . . . . . . 95
7.6. Parallel eﬃciency of Cilksort with weak scaling (1G elements per node). . . . . . . . . . . . . 95
7.7. Proﬁling of Cilksort with weak scaling (1G elements per node). . . . . . . . . . . . . . . . . 96
7.8. Execution time of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)). . . . . . 98
7.9. Execution time of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)). . . . . . 98
7.10. Proﬁling of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)). . . . . . . . . 99
7.11. Execution time of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)). . . . . 100

viii

List of Figures
7.12. Execution time of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)). . . . . 100
7.13. Proﬁling of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)). . . . . . . . 101
7.14. Throughput (number of nodes visited per second) of UTS-Mem with strong scaling. . . . . . . 102
7.15. Parallel eﬃciency of UTS-Mem with strong scaling. . . . . . . . . . . . . . . . . . . . . . . . 102
7.16. Execution time of ExaFMM (particle interactions) with strong scaling. . . . . . . . . . . . . . 105
7.17. Parallel eﬃciency of ExaFMM (particle interactions) with strong scaling. . . . . . . . . . . . 105
7.18. Proﬁling of ExaFMM (particle interactions) with strong scaling. . . . . . . . . . . . . . . . . 106
7.19. Execution time of ExaFMM (particle interactions) with weak scaling. . . . . . . . . . . . . . 108
7.20. Parallel eﬃciency of ExaFMM (particle interactions) with weak scaling. . . . . . . . . . . . . 108
7.21. Proﬁling of ExaFMM (particle interactions) with weak scaling. . . . . . . . . . . . . . . . . 109
7.22. Execution time of ANNS graph construction with strong scaling(bigann-10M). . . . . . . . . 111
7.23. Parallel eﬃciency of ANNS graph construction with strong scaling (bigann-10M). . . . . . . 111
7.24. Proﬁling of ANNS graph construction with strong scaling (bigann-10M). . . . . . . . . . . . 112
7.25. Query per second (QPS) on one node with various recall rates (bigann-10M). . . . . . . . . . 113
7.26. Naive PageRank implementation in Itoyori. . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.27. Execution time of PageRank (three iterations) with strong scaling. . . . . . . . . . . . . . . . 116
7.28. Execution time of PageRank (three iterations) with weak scaling (R-MAT: 𝑁 = 224 ×𝑥, 𝑀 = 16𝑁).117
7.29. Proﬁling of PageRank (three iterations) with strong scaling (twitter2010). . . . . . . . . . . 118
7.30. Proﬁling of PageRank (three iterations) with weak scaling (R-MAT: 𝑁 = 224 × 𝑥, 𝑀 = 16𝑁). . 119
A.1. Cilksort program written with Itoyori in C++. . . . . . . . . . . . . . . . . . . . . . . . . . . 126
A.2. Binary search program with Itoyori in C++. . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
B.1. Execution time of ANNS graph construction with strong scaling (msspacev-10M). . . . . . . . 129
B.2. Parallel eﬃciency of ANNS graph construction with strong scaling (msspacev-10M). . . . . . 129
B.3. Proﬁling of ANNS graph construction with strong scaling (msspacev-10M). . . . . . . . . . . 130
B.4. Query per second (QPS) on one node with various recall rates (msspacev-10M). . . . . . . . . 131
B.5. Proﬁling of PageRank (3 iterations) with strong scaling (rmat26). . . . . . . . . . . . . . . . 132

ix

List of Tables
4.1. Experimental environment for distributed multithreading. . . . . . . . . . . . . . . . . . . . . 42
4.2. Statistics of join and steal events in child stealing and continuation stealing. . . . . . . . . . . 46
4.3. Execution times of LCS on ITO-A with 576 cores (16 nodes). . . . . . . . . . . . . . . . . . . 51
6.1. Experimental environment for shared-memory ADWS (Oakbridge-CX supercomputer [140]).

78

7.1. Experimental environment for Itoyori application studies. . . . . . . . . . . . . . . . . . . . . 89
7.2. Lines of code (LOC) summary for diﬀerent implementations of ExaFMM. . . . . . . . . . . . 104

x1. Introduction
1.1. Background and Motivation
Computer programming has evolved to abstract away low-level hardware details, focusing instead on higherlevel algorithms and logic. However, even today, low-level programming is still required for high-performancecomputing (HPC) on massively parallel computers, including supercomputers. Such large-scale systems
typically consist of multiple machines (i.e., compute nodes), forming distributed memory, in which eachmachine has its own view of memory. In distributed-memory computing, data are exchanged between
machines through communication over network interconnects. Message Passing Interface (MPI) [6] is astandard programming interface for communication in HPC. A common practice in MPI programming is for
programmers to take full control of load balancing—they explicitly program which machine performs which
computation and how machines communicate with each other. This means that application programmers needto explicitly specify how parallel computation should be done. Ideally, however, it would be preferable if they
only needed to describe what needs to be computed in parallel.
Furthermore, today, each machine typically has a multi-core CPU with shared memory, allowing multiple
cores to share the same view of memory. In such environments, it is often more eﬃcient to exchange data via
shared memory rather than performing explicit communication between cores. This leads to the adoption ofshared-memory parallel programming models, such as OpenMP [7]. As a result, today’s HPC programming
requires the use of two distinct programming models by explicitly distinguishing between inter- and intra-nodeenvironments: MPI for distributed memory across machine boundaries and OpenMP for shared memory
within each machine. In other words, to solve a single problem, two types of programming models are used
for diﬀerent parallelization schemes at two levels. Although this approach can maximize performance if doneby HPC experts, it lacks productivity.
This situation arises from the challenges in designing programming models and their runtime implementations that are highly productive while simultaneously achieving high performance. We consider thattask-parallel programming is a promising approach to this issue. In task-parallel programming, parallel computations are expressed in terms of logical series-parallel relationships between tasks. Tasks have dependencies
among them, which form a directed acyclic graph (DAG). In dynamic task parallelism (fork-join parallelism),
any task can dynamically fork new tasks at any points of execution. This means that the task graph needs not tobe pre-computed before execution. In addition, programmers can create a massive number of tasks, regardlessof actual hardware parallelism (e.g., the number of cores). This is enabled by the underlying runtime system,
which automatically schedules tasks to actual hardware. Hence, the task-parallel model is well suited for expressing dynamic and irregular parallelism, which is often found in tree-based computation, graph processing,and so on.
The task-parallel model allows programmers to concentrate only on logical parallelism, while keeping
suﬃcient ﬂexibility for expressing dynamic and irregular computations. Because of these beneﬁts, manyeﬃcient task-parallel runtime systems have been developed until now, showing great scalability to manycore CPUs with tens or hundreds of cores. A representative example is Cilk [8], [9], and subsequently,

11. Introduction
language extensions or libraries for C/C++ such as OpenMP [7], OpenCilk [10], oneTBB [11] (formerlyIntel TBB [12]), Microsoft Parallel Patterns Library (PPL) [13], Tascell [14], and MassiveThreads [15] haveemerged and supported task parallelism. In addition, dynamic task parallelism has been integrated into various
high-level languages, either as a built-in feature of the language or through libraries: Chapel [16], X10 [17],Java Fork/Join Framework [18], Go [19], Julia [20], OCaml Domainslib [21], Task Parallel Library (TPL) [22]for .NET languages such as C# and F#, and extensions to the Standard ML, including SML# [23] and MPL [24],[25]. Nevertheless, in the context of distributed memory, there are very few systems that automatically schedule
dynamic tasks across machine boundaries.

1.2. The Itoyori Runtime System and Its Challenges
This dissertation presents how to eﬃciently scale dynamic task parallelism beyond machine boundaries. We
demonstrate this by developing Itoyori1 , a distributed task-parallel runtime system. While many existing
runtime systems share a similar goal, Itoyori ambitiously seeks to achieve both productivity and performanceat a high level. The high productivity of Itoyori is derived from its simple and intuitive programming model,
which integrates global task parallelism—in which tasks are automatically load-balanced across machineboundaries—and global address space—in which physically distributed data can be uniformly addressed andaccessed from any machines.
Global address space is a view of shared memory over distributed-memory machines, eliminating the needfor explicit inter-node communication. This is particularly useful for global task parallelism because tasks
can be executed on any compute nodes. Speciﬁcally, Itoyori employs the Partitioned Global Address Space(PGAS) model for global address space. In the PGAS model, memory access to the global address space is
automatically translated into communication with remote nodes. Therefore, global memory access is usuallymore expensive than local memory access. In general, the PGAS model requires programmers minimize data
movement between remote and local memory by explicitly distinguishing global and local memory access.
However, naively combining these two concepts—the global task-parallel model and the PGAS model—
easily results in poor performance due to bad data locality. In general, to enhance data locality, it is necessary
either (1) to keep the data in memory closer to the processors (e.g., cache memory), or (2) to move computations
to where the data are located. However, a naive combination of existing PGAS and task-parallel systems makes
either approach challenging. In the following, we outline the desired approaches for enhancing data locality in
task-parallel systems and the reasons why they are challenging.
1. Enhancing data locality by running multiple tasks that are close in the task graph on the same processor
and storing their data in nearby memory. This approach is considered beneﬁcial because tasks that areclose in the task graph typically share the same data. It is known that close tasks can be scheduled
to the same processor using the work stealing scheduler [26], [27], and their data are transparently
kept in cache memory if hardware support exists. However, today’s typical PGAS implementations
lack such implicit caching mechanisms. Instead, they require that programmers explicitly copy remote
data to local memory in large chunks. Yet, integrating this with task parallelism poses a challenge in
pre-copying the data shared by multiple tasks, as the execution locations of tasks are determined atruntime. Consequently, each task initiates communication individually for its own data, resulting inredundant and ﬁne-grained communication.
1 Itoyori is the Japanese name of the ﬁsh “threadﬁn breams.” The Itoyori runtime system is being developed at https://github.

com/itoyori/itoyori.

2

1. Introduction
2. Enhancing data locality by scheduling tasks so that data movement is minimized by considering data
placement and memory hierarchies. However, the commonly used work-stealing scheduler, which
randomly selects victims for stealing tasks, does not take into account data placement or memory
hierarchies at all. Some existing schedulers consider them to some extent, but to the best of ourknowledge, none can uniformly handle deep memory hierarchies, ranging from hierarchical caches ofmodern CPUs and Non-Uniform Memory Access (NUMA) architectures to distributed memory.
In this dissertation, we address each of the identiﬁed problems about data locality by the following solutions:
1. Introducing a software cache to the PGAS system to implicitly fetch larger data than requested and reusepreviously fetched data in local memory. This enables aggregating communication for the data shared
among multiple tasks that are scheduled on the same processor.
2. Devising a novel task scheduler Almost Deterministic Work Stealing (ADWS) that matches the taskhierarchy and the memory hierarchy. ADWS considers both eﬃcient cache reuse on hierarchicalhardware caches and data placement on NUMA architectures and distributed memory.
Itoyori implements both techniques, achieving high eﬃciency and scalability due to enhanced data locality.
In the next section, we explain our contributions in more detail.

1.3. Overview and Contributions of This Dissertation
In this section, we provide an overview of this dissertation. Initially, we explain the background and preliminaries in Chapter 2 before discussing our speciﬁc contributions. In the following, we brieﬂy summarize ourcontributions in subsequent chapters.
Chapter 3 gives an overview of the Itoyori runtime system and introduces its programming model. Itoyori
is implemented as a C++17 header-only library, and its C++ APIs are explained here. Speciﬁcally, we madethe following contributions in Chapter 3:• We proposed new APIs for global memory access, called checkout/checkin APIs. They are designed to
avoid copy semantics between local and global memory, unlike conventional GET/PUT APIs in PGAS
systems. This is beneﬁcial both in performance and programmability.
• We designed high-level C++ interfaces that resemble the C++ standard (e.g., C++17 parallel STL), sothat the checkout/checkin semantics is properly incorporated. Using these high-level interfaces, most of
the raw fork/join and checkout/checkin calls can be omitted from user programs.
Chapter 4 discusses the implementation of ﬂexible, yet highly scalable multithreading on distributed memory, which is the basis of global task parallelism in Itoyori. Continuation stealing is a common work-stealing
technique in Cilk and other shared-memory task-parallel systems, but this is not popular on distributed memory
because it involves thread stack migration across diﬀerent machines. Previously, the uni-address scheme [28],[29] has been proposed for eﬃcient continuation stealing on distributed memory, but no performance comparison with other threading techniques has been performed. We made improvements to the uni-address scheme
and conducted comprehensive performance studies to verify its practicality. As it demonstrated great eﬃciency
and ﬂexibility, we employed the uni-address scheme as the Itoyori’s multithreading layer. In the following, welist our contributions about distributed multithreading:31. Introduction• We introduced techniques to eﬃciently synchronize (join) threads by utilizing Remote Direct Memory
Access (RDMA) atomic operations.
• We experimentally demonstrated that ﬂexible continuation stealing can be realized on distributed memory
with little overhead by comparing to other work-stealing techniques and existing systems. Even more,
our results suggested that continuation stealing had overall beneﬁts for complicated problem settings.
• We conducted a large-scale experiment with the unbalanced tree search (UTS) benchmark and demonstrated 96.4% parallel eﬃciency on 110,594 cores (2,304 nodes). Although the UTS benchmark does not
involve any global memory access, this shows great eﬃciency and scalability of global task parallelism.
The next two chapters introduce the primary contributions of this dissertation: data locality improvements
for global address space and global task parallelism.
Chapter 5 explains the design and implementation of Itoyori’s software cache for global memory access. We
ﬁrst explain the software cache implementation for checkout/checkin APIs, which is not necessarily speciﬁc
to task parallelism. We then discuss eﬃcient integration with fork-join task parallelism, speciﬁcally the one
with the work-stealing scheduler. Summarizing our contributions in Chapter 5:• We devised a ﬁxed-size, private cache implementation for checkout/checkin APIs. This involves dynamic
updates to virtual memory mappings, as the checkout/checkin APIs necessitate the use of the same virtual
address for accessing both global and local memory.
• We introduced eﬃcient RDMA-based cache coherence management that adheres to the work-ﬁrst principle [9] of work stealing. This principle suggests that for eﬃcient work-stealing implementations, theoverhead at each fork/join should be moved to the less frequent work-stealing events. As such, we delay
costly coherence actions (e.g., cache invalidation and write-back) until work-stealing events occur.
Chapter 6 introduces Almost Deterministic Work Stealing (ADWS), a locality-aware task scheduler for deep
memory hierarchies. Note that ADWS is not speciﬁc to Itoyori or distributed memory; it can be beneﬁcial
even for deep memory hierarchies on shared memory, such as hierarchical caches and NUMA architectures. In
Chapter 6, we explain both shared-memory and distributed-memory implementations of ADWS. Speciﬁcally,
our contributions are:
• We proposed ADWS to improve data locality of fork-join task-parallel programs in deep memory
hierarchies. ADWS ﬁrst performs deterministic task mapping to improve initial data locality by matching
the task hierarchy and the memory hierarchy. It then performs dynamic load balancing to ﬁx load
imbalance by stealing tasks from as close cores as possible.
• We extended ADWS to multi-level ADWS to avoid excessive capacity misses for shared caches. It
schedules tasks so that only tasks whose working set sizes are small enough to ﬁt into a shared cache are
simultaneously scheduled within the cores sharing the cache.
• We evaluated ADWS on shared memory by implementing it in MassiveThreads [15] and demonstrated
its beneﬁts for hierarchical caches and NUMA architectures. Our results show that ADWS is evenbeneﬁcial on a single machine (out of the context of Itoyori).

4

1. Introduction
Chapter 7 evaluates the Itoyori runtime system using various benchmarks and real-world applications, including sorting (Cilksort), stencil computation (Heat), tree search (UTS-Mem), N-body simulation (ExaFMM),
graph construction for approximate nearest neighbor search (ANNS), and graph processing (PageRank). Allof our benchmarks and applications have been straightforwardly ported from shared-memory task-parallel
implementations to Itoyori without major changes to their overall parallel algorithms. Our ﬁndings from ourevaluation are summarized as follows:• Software caching is beneﬁcial when the program exhibits a high degree of spatial or temporal locality.
The performance improvement was particularly highlighted for ﬁne-grained parallelism, in which data
sharing among tasks is challenging without runtime-level caching. A notable example is ExaFMM,
which showed up to 9.8× faster performance by introducing software caching.
• ADWS is beneﬁcial particularly for applications with relatively regular parallelism (i.e., well-balanced
task hierarchy) and regular memory access patterns. For example, in Cilksort, ADWS was up to 2.2×
faster than random work stealing. The Heat benchmark beneﬁts the most from ADWS, achieving over
95% parallel eﬃciency on 576 nodes (27,648 cores) for weak scaling.
• Many applications could be scaled to distributed memory with Itoyori, but its parallel eﬃciency depends
on many factors, such as the amount of logical parallelism, computation to memory access ratio (bytes
per FLOPS), and memory access locality. Overall, if parallel applications have suﬃcient locality andparallelism, Itoyori can eﬃciently run them on distributed memory with high scalability. Notably,
ExaFMM showed a 10× speedup for strong scaling from a single node to 12 nodes, and maintained 75%
parallel eﬃciency on 512 machines for weak scaling.
• Itoyori programs performed competitively to MPI+OpenMP implementations in most cases (Cilksort,Heat, ExaFMM, PageRank) on distributed memory. Exceptionally, for strong scaling on many nodes inthe Heat benchmark, Itoyori substantially underperformed MPI+OpenMP.
• Although it was straightforward to port existing shared-memory algorithms to Itoyori, not all algorithms
were successfully scaled to distributed memory. In particular, for highly irregular applications withrandom memory access (e.g., graph processing), Itoyori performs poorly on distributed memory. They
are the cases where the locality-related assumptions made by Itoyori were not true.
Then, we introduce more related work in Chapter 8 and draw conclusions in Chapter 9.

5

2. Background and Preliminaries
In this chapter, we present the background and preliminaries of this dissertation. First, we introduce thehardware conﬁgurations of modern CPU-based systems, including both shared-memory and distributedmemory architectures, in Section 2.1. Then, we explain the models and implementations of task parallelism
and global address space in Section 2.2 and Section 2.3, respectively.

2.1. Modern Architectures of Multi-Core CPUs and Clusters
This section explains typical conﬁgurations of recent CPU architectures, which we will also use in ourevaluation (Chapter 7). Our primary focus in this section is on deep and complex memory hierarchies ofmodern CPUs. The memory hierarchy of modern CPUs is complicated by hierarchical caches (Section 2.1.1)
and Non-Uniform Memory Access (NUMA) architectures (Section 2.1.2). Furthermore, distributed memoryof compute clusters is also considered a part of the memory hierarchy. In such large-scale systems, multiple
CPU nodes are connected with high-performance interconnects with the capability of Remote Direct Memory
Access (RDMA) (Section 2.1.3).

2.1.1. Hierarchical Caches
Typically, CPUs implement hardware caches to transparently exploit the locality of reference in programs. It is
well-known that programs often exhibit two types of locality: temporal locality and spatial locality. Temporal
locality indicates that if a memory location is referenced, the same location is likely to be accessed again in thenear future. Spatial locality means that if a memory location is referenced, nearby locations are likely to beaccessed in the near future. To exploit these two types of locality, recently accessed data are temporarily placedon a fast cache by replicating the data at the main memory. At this time, the data are fetched at a cache-line
granularity, which is typically larger than the actual memory access granularity. Cache memory is often
conﬁgured hierarchically and commonly named L1, L2, and L3 cache in the order of closeness to the core.
In general, faster but smaller-capacity caches are placed close to the cores, while slower but larger-capacity
caches are positioned close to the main memory.
48 cores38 cores
Core

Core

Core

Core

Core

Core

Core

Core

Core

12 cores
Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

L1

32 KB

L2

L2

L2

L2

L2

L2

L2

L2

L2

L2

1.25 MBL3: 57 MB

L3: 57 MB

DRAM (NUMA node)

DRAM (NUMA node)

SocketSocket

L2: 8 MBL2: 8 MBDRAM (NUMA node)

DRAM (NUMA node)Core Memory Group (CMG)

Core Memory Group (CMG)
Socket(a) Intel Xeon Platinum 8368.

(b) Fujitsu A64FX.

Figure 2.1.: Memory hierarchies of modern CPU architectures.

6

64 KB

2. Background and Preliminaries
Multi-core CPUs often have shared caches, which are shared by multiple cores. On the other hand, a cachethat is dedicated to a single core is called a private cache. Modern CPUs typically feature both private andshared caches. Figure 2.1 shows actual memory hierarchies of CPUs that we use in our evaluation (Section 7.1).
For example, the Intel Xeon Platinum 8368 CPU has private L1 and L2 caches and a shared L3 cache for coresin a single chip (socket). In a multi-socket conﬁguration, where a compute node has multiple sockets, there
are multiple shared caches, each shared among a subset of cores. In contrast, the Fujitsu A64FX CPU has four
partially shared caches within a single socket. Although the speciﬁcs of actual conﬁgurations vary, with theincrease in the number of cores, the cache hierarchy tends to become more complex, often featuring partiallyshared caches.
Cache Coherence

Having hierarchical caches in multi-core CPUs, the data at the same memory location can exist duplicated at
multiple caches at diﬀerent hierarchy levels. Even in such a situation, we need to ensure cache coherence forthe cached data. Consider two cores A and B have separate private L1 caches, and the value (0) at the same
memory location X is cached in both L1 caches. Then, core A updates the value (0 → 1) at X, which is thencached in its own L1 cache. After that, core B reads a value at X. At this time, one might expect that the
updated value (1) is passed from core A’s L1 cache to core B’s L1 cache, but without a special mechanism,
core B would continue reading a stale value (0) indeﬁnitely.
According to the deﬁnition by Nagarajan et al. [30], cache coherence is deﬁned in terms of the following
two invariants:
1. Single-Writer, Multiple-Read (SWMR) Invariant. For any memory location A, at any given time, there

exists only a single core that may write to A (and can also read it) or some number of cores that may
only read A.
2. Data-Value Invariant. The value of the memory location at the start of an epoch is the same as the valueof the memory location at the end of the its last read-write epoch.
An epoch for a memory location is a time period during which either a single core has read-write access, or
multiple cores (possibly none) have read-only access to that memory location. To interpret these two invariants
simply, there can only be one writer for a memory location at any given time, and the last write by the writer
must be correctly read by the subsequent readers.
A cache coherence protocol ensures that these two invariants hold for each memory location. Typically, CPUcaches employ the MSI protocol or its variants (e.g., MESI, MOESI). In the following, we brieﬂy introduce
how the MSI protocol maintains the SWMR and data-value invariant. In the MSI protocol, each cache line is in
one of the three states: Modiﬁed (M), Shared (S), or Invalid (I). The “M” state corresponds to the single-writerepoch in the SWMR invariant. While a cache is holding a cache line with the “M” state for a memory location
X, no other cache can have the “M”-state cache line at X. Therefore, only the “M”-state cache line can update
its value. On the other hand, the “S” state corresponds to the multiple-reader epoch in the SWMR invariant.
Multiple caches can have the “S”-state cache line at X at the same time, and they have the same value at X. By
properly managing transitions between these three states, the SWMR invariant is maintained.
The data-value invariant is maintained by appropriately performing write-back for modiﬁed (dirty) caches
and invalidation for shared (read-only) caches. In order to write to an “S”-state cache line, it needs to ﬁrst
change its state to the “M” state, invalidating all other caches (changed to the “I” state). This ensures that this7

2. Background and Preliminaries
cache line’s value can be exclusively updated by its local core. When another core requests a read from an“I”-state cache line, it sends a write-back request to the cache that has the corresponding “M”-state cache line
(if any). Then, the target cache writes back its modiﬁed data to the main memory and changes the cache line
state from the “M” state to the “S” state. After that, on the requester side, the “I”-state cache line transitions
to the “S” state by fetching the latest value from the main memory. This ensures that duplicated cache lines atthis memory location are all up-to-date.
The above explanation is over simpliﬁed, and even in a single-level cache, cache coherence protocols canbecome quite complex in practice. In reality, extended versions of the MSI protocol involve numerous statetransitions and require careful considerations for concurrent accesses. Furthermore, ensuring cache coherencein hierarchical caches adds another layer of complexity. In many cases, diﬀerent cache coherence protocols
are employed at each level of the hierarchy, and the system is cleverly designed to maintain cache coherence
as a whole.
Note that cache coherence does not specify the memory ordering between diﬀerent memory locations. The
memory ordering of possibly concurrent read and write accesses to multiple memory locations is governed bymemory consistency. A memory consistency model deﬁnes the rules for memory ordering in a view of sharedmemory. Memory consistency models are deﬁned at the programming language level or at the Instruction
Set Architecture (ISA) level of CPU architectures. Cache coherence is merely a means to implement memory
consistency models and usually not visible to programmers. In this dissertation, Chapter 3 explains Itoyori’s
memory consistency model and Section 5.3.4 explains the cache coherence protocol to ensure the memory
consistency model.
Programming with Hierarchical Caches

Typical CPU caches are transparent to programmers—meaning that memory accesses to virtual memory
addresses are seamlessly cached by the hardware, without involvement of programmers. That is, programmers
do not have to explicitly move data from the main memory to cache memory to beneﬁt from faster hardware
caches. Nevertheless, at the same time, programmers are required to write programs with a high degreeof locality to achieve high performance. This may involve designing cache-friendly data structures to avoid
random memory access, or limiting the working set size to ﬁt into the cache capacity. In addition, regularmemory access patterns (e.g., sequential memory access) will beneﬁt from the hardware prefetcher, which
speculatively moves data from the main memory to cache memory ahead of time.
To better utilize CPU caches, programmers often employ cache-aware algorithms, which explicitly incorporate cache capacities or cache-line sizes into the algorithms. For example, cache blocking is a techniquethat explicitly rearranges the memory layout so that each subset of the data will ﬁt into the cache. Typically,an algorithm is decomposed into subproblems so that each subproblem ﬁts within the cache capacity. Then,these subproblems are computed one by one for good cache eﬃciency. However, this approach can easily leadto hardware-speciﬁc and nonportable programs, as it assumes a speciﬁc cache conﬁguration. Moreover, it puts
a considerable burden on programmers, especially when computing on deep and complex cache hierarchies
in modern CPUs. In contrast, cache-oblivious algorithms [31], [32] do not depend on any particular cacheconﬁgurations, yet achieving analytically good cache behaviours for arbitrary cache conﬁgurations. Typicalcache-oblivious algorithms rely on recursive problem decomposition, which aligns well with fork-join task
parallelism explained in Section 2.2.

8

core

core

core

core

core

core

core

core

core

core

core

core

SocketDRAM

DRAM

2. Background and Preliminaries

SocketRemote Memory Access

Local Memory AccessFigure 2.2.: Local and remote memory access in Non-Uniform Memory Access (NUMA) architectures.

2.1.2. Non-Uniform Memory Access (NUMA) Architectures
Apart from hierarchical caches, modern CPUs often have a diﬀerent type of memory hierarchy: Non-Uniform
Memory Access (NUMA) architectures. NUMA systems typically provide a uniform shared-memory viewat the hardware level, but the actual memory access cost is non-uniform. That is, the memory access costdepends on the distance between the cores and the physical memory locations. This is because NUMA
architectures have multiple, distributed Dynamic Random Access Memory (DRAM) modules. The purposeof this distribution is to reduce contention to DRAM and improve overall throughput, as the number of CPU
cores increases. Local memory access refers to accessing the DRAM module that is local to the initiating CPU
core, while remote memory access refers to accessing non-local DRAM modules. Figure 2.2 illustrates local
and remote memory access in NUMA architectures. In this ﬁgure, each socket is attached a separate DRAM
module. Thus, access to local DRAM (local memory access) is much faster than that to remote DRAM (remotememory access). A NUMA node refers to a group of cores that share the same local DRAM. In Figure 2.2, asocket corresponds to a NUMA node, but this is not always the case. For example, the Fujitsu A64FX CPU(Figure 2.1) has multiple NUMA nodes within a single socket.
While hierarchical caches are intended to temporarily hold copies of data from the main memory, NUMA
architectures are designed to hold the original data across distributed memory instances. Once a physicalpage is settled in a NUMA node, the page remains in the same NUMA node unless explicitly relocated byprogrammers (via system calls) or implicitly moved by the OS. Therefore, for high performance, it is importantto place physical pages at appropriate NUMA nodes ahead of time and compute on the NUMA nodes that havethe original data.
The numactl command is a common tool to easily conﬁgure the NUMA memory placement policy. Bydefault (the --localalloc/-l option), physical pages are allocated from the local NUMA node (following theﬁrst-touch policy in Linux). However, this is problematic in scenarios such as when a single thread initializes alarge array, and then multiple threads compute on it in parallel. In such cases, the entire array will be allocatedto a single NUMA node, limiting overall memory throughput. The --interleave=all/-iall option canbe used to address this issue, as it tells the system to evenly distribute physical pages to all NUMA nodes.
Although the page placement is almost random, the overall memory throughput can be improved. To explicitly
specify the NUMA node locations for each page, programs need to be modiﬁed using the libnuma library orproper system calls. Note that Linux supports automatic NUMA balancing [33], which automatically relocatesphysical pages to the NUMA nodes that have frequently accessed the pages in a certain period. However,relocation takes a considerable amount of time and may not work well for irregular programs.
NUMA architectures can also be considered as distributed-memory computers, as the memory is physically
distributed. However, they are generally categorized as a type of shared-memory computers. This is because theCPU hardware provides a shared-memory view—programs can transparently access the memory in diﬀerent

92. Background and Preliminaries

User Application
MemoryOS

User Application
MemoryMemoryMemorySockets

Sockets

TCP

TCP

IP

RDMA

NIC

IP

Interconnect

OS

NIC

Figure 2.3.: Conventional socket-based communication over TCP/IP and Remote Direct Memory Access(RDMA) technology that bypasses the kernel.
NUMA nodes and beneﬁt from coherent caches over them. Nevertheless, the memory access cost is still
diﬀerent between local and remote memory access, requiring consideration of memory distribution and task
placement on the program side. This requirement is not fundamentally diﬀerent from the common practices
in distributed-memory computing. From another perspective, if a shared-memory view and coherent caches
are provided for distributed-memory computers at the software level, they can be seen as a natural extension
to NUMA architectures. This is the concept behind Itoyori’s approach of unifying shared- and distributedmemory models.

2.1.3. High-Performance Interconnects and Remote Direct Memory Access (RDMA)
Large-scale distributed-memory computing systems, such as clusters and supercomputers, are equipped withhigh-performance network interconnects, such as InﬁniBand [34] and Tofu Interconnect D [35]. Interconnects
link compute nodes through their Network Interface Cards (NICs), oﬀering low-latency and high-throughput
communication. Nevertheless, the latency of high-performance interconnects remains in the range of several
microseconds, two orders of magnitude greater than tens of nanoseconds required for DRAM access [36].
In contrast, the bandwidth of recent interconnects is rapidly increasing, so trading oﬀ bandwidth for reduced
latency is considered a viable approach [37]. In principle, data should be transferred in as large a granularity as
possible, which is a common practice in MPI programming. To better utilize high bandwidth of interconnects,
data structures and memory layouts are often tweaked so that data buﬀers for communication are stored incontiguous memory.
Remote Direct Memory Access (RDMA) is a key technology in modern high-performance interconnects.
Before introducing RDMA, we ﬁrst examine the case of using standard sockets for communication. Standard
socket communication is handled by the OS kernel, which involves context switching between the user andkernel spaces, user-kernel data copies, and packet processing for protocol stacks such as TCP/IP. These
processes consume many CPU cycles and result in high latency. Furthermore, socket communication requires
the CPU involvement on the target node (two-sided communication). That is, if a node wants to fetch datafrom a remote node, it needs to interrupt the remote node to initiate two-sided communication. Figure 2.3
depicts this situation, in contrast to RDMA.
RDMA oﬀers kernel-bypassing, low-latency, and one-sided communication. RDMA enables initiating
communication directly from the user space, bypassing the kernel and its heavyweight operations. Data
are directly copied between local and remote memory, without creating copies using any additional buﬀers

10

2. Background and Preliminaries(zero-copy communication). Furthermore, read and write operations to remote memory can be performed
without any involvement of the target CPU (one-sided communication). As illustrated in Figure 2.3, data canbe directly transferred between local and remote memory through NICs and interconnects.
To initiate RDMA between local and remote memory, both memory must be pre-registered to NICs andpinned to the DRAM. This process is known as memory registration. Once registered, the memory can beaccessed via RDMA using either handlers (unique IDs) assigned during the registration process or virtual
addresses, depending on the speciﬁcs of interconnects. Typically, memory registration involves the OS kernel
operation.
Major MPI implementations, such as MPICH [38] and Open MPI [39], utilize RDMA for point-to-point
communication (e.g., MPI Send() and MPI Recv()) and collective operations. However, due to their interface
design, these operations are either two-sided or many-to-many communication, which cannot leverage one of
the major advantages of RDMA: asynchronous, one-sided communication. Therefore, MPI-3 Remote Memory
Access (RMA) [40] has been introduced to the MPI standard as interfaces for one-sided communication. In
MPI-3 RMA, programmers can initiate one-sided communication for reading from remote memory withMPI Get() and writing to remote memory with MPI Put(). As these GET/PUT calls are nonblocking, theircompletion can be ensured by the MPI Win flush() call or its variants (in the passive mode). Memory
registration corresponds to the creation of windows in MPI-3 RMA (e.g., MPI Win create()).
Note that, although MPI-3 RMA is designed with RDMA in mind, its implementations are not required touse RDMA for the sake of portability. In fact, the study by Schuchart et al. [41] shows that, even in the presenceof RDMA-capable interconnects, MPICH-based implementations did not directly oﬄoad RMA operations to
RDMA, whereas Open MPI did. In cases where RDMA is not utilized, the RMA implementation wouldinternally involve two-sided communication, meaning that it is not truly one-sided. In other words, one-sided
communication in such scenarios cannot be completed until the target node participates in the communication
process (e.g., by invoking MPI calls to make progress). For this reason, we will use Open MPI as the MPI-3
RMA implementation in our evaluation.
In addition, modern interconnects often support remote atomic operations, such as compare-and-swap (CAS)and fetch-and-add (FAA), for handling concurrent access to the same memory location. Accordingly, MPI-3
RMA also supports one-sided atomic operations such as MPI Compare and swap() and MPI Fetch and op().
However, it depends on speciﬁc MPI implementations whether these atomic calls are directly oﬄoaded to the
RDMA atomic operations supported by the underlying interconnects.
MPI-3 RMA is a well-established standard for one-sided communication, but this is not the only interface
for RDMA. Until now, many communication libraries and Partitioned Global Address Space (PGAS) systems
have been developed as a means to leverage one-sided communication and RDMA. More details of PGAS
systems will be covered in Section 2.3.2.

2.2. Task Parallelism and Work Stealing
The previous section has explained architectural details of today’s complicated shared- and distributed-memory
CPU-based systems. To fully utilize these hardware resources, programmers would be required to design
cache-friendly parallel algorithms on complicated hierarchical caches, consider memory distribution and task
placement in NUMA architectures and distributed memory, specify communication patterns between multiple
compute nodes, rearrange the memory layout for high-throughput communication over high-performanceinterconnects, and so on. These manual tuning eﬀorts, while often necessary to achieve the system’s peak11

2. Background and Preliminaries
thread th1 = fork([]{ A(); });thread th2 = fork([]{ B(); });thread th3 = fork([]{ C(); });D();
5 th1.join();6 th2.join();7 th3.join();1
2
34(a) Threads (futures).

task_group tg;
tg.run([]{ A(); });tg.run([]{ B(); });4 tg.run([]{ C(); });5 D();
6 tg.wait();123(b) Task group.

parallel_invoke([]{ A(); },3[]{ B(); },4[]{ C(); },5[]{ D(); });12(c) parallel invoke().

Figure 2.4.: Typical task-parallel APIs in C++ libraries. In all cases, executions of functions A(), B(), C(),and D() are logically parallel.
performance, can signiﬁcantly diminish productivity.
Instead, this dissertation focuses on task-parallel programming, which is expected to achieve a better balance
between productivity and performance. In this section, we ﬁrst explain the task-parallel model in Section 2.2.1
and then the work-stealing scheduler in Section 2.2.2. While task-parallel runtime implementations vary,
Section 2.2.3 describes implementations using user-level threads, which are suﬃciently ﬂexible and available
with ordinary C/C++ compilers. Then, Section 2.2.4 discusses data locality in task-parallel programs, how
work stealing exploits it, and challenges in deep memory hierarchies. Finally, Section 2.2.5 introduces existing
distributed-memory task-parallel runtime systems.

2.2.1. Fork-Join Task Parallelism
The deﬁnition of task parallelism varies across literature. A common consensus is that the task-parallelmodel is an execution model based on parallel tasks and their dependencies, forming a directed acyclic graph(DAG). In particular, if a DAG is provided in advance, it is often referred to as static task parallelism, whereas if
tasks are dynamically generated, it is commonly called dynamic task parallelism. This dissertation focuses on
dynamic task parallelism because of its generality and capability to deal with dynamic and irregular algorithms.
Fork-join task parallelism, or simply fork-join parallelism, is a form of dynamic task parallelism. A fork
operation creates a new task, which may run in parallel to the continuation of the current task. A join operation
joins one or more tasks, blocking the current task execution until their completion. Fork-join parallelism is
simply expressed by these fork and join operations. Figure 2.4 shows multiple options to express fork-joinparallelism using task-parallel library APIs in C++. In these examples, four functions (A, B, C, and D) areintended to be executed in parallel. The low-level thread (or future) primitive (Figure 2.4a) returns a thread
handler when forking a thread, and the thread is joined by using the handler. The last computation (D) can also be
forked as a new task, but not required to do so. The task group notation (Figure 2.4b) and parallel invoke()(Figure 2.4c) call are oﬀered by task-parallel libraries such as oneTBB [11] and Microsoft PPL [13]. A task
group can dynamically fork any number of tasks and join all tasks within the group at once. In contrast,
parallel invoke() forks a ﬁxed number of tasks and joins them at once. The parallel invoke() call is
the simplest notation, but it is not suited for the cases where the number of tasks is not known in advance.
As a running example, we consider fork-join parallelization of the recursive merge sort algorithm. Figure 2.5
shows its C++ code written with the parallel invoke() call. The mergesort() function receives asarguments a pointer and a size, representing a contiguous memory region. The input region [𝑎, 𝑎 +𝑛) is divided
in the middle, and each of the two subregions is sorted recursively. The computations of sorting two subregions
can be executed in parallel, and thus they are spawned as two parallel tasks with parallel invoke(). After

12

2. Background and Preliminaries
1
2
3
4
5
6
7
8
9
10
11

template <typename T>
void mergesort(T* a, size_t n) {if (n < cutoff) {std::sort(a, a + n);} else {parallel_invoke([=] { mergesort(a
, n/2
); },
[=] { mergesort(a + n/2, n - n/2); });merge(a, n/2, n); // Omit the internal implementation of merge()}}

Figure 2.5.: Recursive parallel merge sort program in C++ with parallel invoke().
joining these two tasks, two sorted subregions are merged. Parallel tasks are recursively created until theregion becomes suﬃciently small. The cutoﬀ size determines the minimum size to further parallelize tasks,
thus controlling the granularity of tasks. At leaf tasks with regions smaller than the cutoﬀ size, it switches tothe sequential std::sort() function.
This example, which forms recursive fork-join parallelism, can dynamically create a massive number oftasks, far exceeding the number of physical cores. Scheduling of these tasks is automatically performed by
the underlying runtime system, freeing programmers from the burden on manual load balancing. This enables
processor-oblivious programming, forgetting about actual hardware details. In addition, it is a clear advantageof fork-join parallelism that parallel constructs can be arbitrarily nested. Because of this property, fork-joinparallelism aligns well with divide-and-conquer algorithms, including merge sort.
To clarify the scope of this dissertation, we deﬁne the fork-join model that this dissertation deals with. In
this dissertation, we say a system supports fork-join parallelism iﬀ it meets all of the following conditions(Conditions 1–4).
Condition 1. The program execution begins with only one root task.
Condition 2. Any task can dynamically fork new tasks at any points of execution.
Condition 3. Every non-root task is joined exactly once by its immediate parent task that forked it.
Condition 4. Every task joins its child tasks in the reverse order of their forking. If multiple tasks are joinedat once, the order between these tasks does not matter.
Condition 2 excludes parallel models in which only the root task can create child tasks. Condition 3 meansthat the lifetime of a task cannot exceed that of its parent, and the parent must join all of its child tasks
before completion. This condition excludes async-ﬁnish parallelism in X10 [17], for example. In addition,
Condition 4 is needed to ensure that parallelism is well structured and properly nested. This deﬁnition offork-join parallelism belongs to a class of fully strict computations [26], and we believe this deﬁnition covers
most of the existing fork-join constructs (e.g., spawn-sync in Cilk [8] and task and taskwait clauses in
OpenMP [42]).
The thread (future) primitive (Figure 2.4a) and the task group primitive (Figure 2.4b) can express forkjoin parallelism, but they cannot guarantee fork-join parallelism under our deﬁnition. Figure 2.6 shows a
program example that does not form fork-join parallelism using the thread primitive. In this example, threads
th1 and th2 are joined in the same order as they were forked, which contradicts Condition 4. In contrast,13

2. Background and Preliminaries

thread th1 = fork([]{ A(); });B();
thread th2 = fork([]{ C(); });th1.join();5 D();
6 th2.join();
1
2
3
4

A

C

1
2
3
4

B

5
6
7
8

D

Figure 2.6.: Example of task parallelism that is notfork-join.
Task (thread)cilk_scope {cilk_spawn A();
cilk_scope {cilk_spawn B();
C();
}
D();
}

A
B

C
D

Figure 2.7.: Example of fork-join parallelism
expressed in OpenCilk [10].

T1

Fork edge
T2

Continue edge
T5

T3

T9

Strand

T1

T7
T4

T6

T8

T10

T2

T11

T4
Join edge

T5
T6

T9
T7
T8

Figure 2.8.: Directed acyclic graph (DAG) of forkjoin parallelism.

Figure 2.9.: Task tree of fork-join parallelism at a certain point in time.

parallel invoke() (Figure 2.4c) guarantees fork-join parallelism, but this is limited to a ﬁxed number oftasks. A more ﬂexible construct to enforce fork-join parallelism is the task scope. OpenCilk [10] has thecilk scope construct to represent task scopes as a language extension to C/C++. Figure 2.7 shows example
OpenCilk [10] code with nested task scopes. A lexical task scope is created by the cilk scope keyword, andchild tasks can be dynamically forked by specifying the cilk spawn keyword. Then, all child tasks are joined
when the innermost task scope ends. In this code, A() is spawned in the outer scope of cilk scope. Then,
B() is spawned in the inner scope in the same task, running concurrently to C(). The task for B() is joined at
the end of the inner scope, before D() begins. The task for A() is joined at the end of the outer scope, afterD() ﬁnishes. This join ordering satisﬁes Condition 4.
Task graphs are also shown in Figure 2.6 and Figure 2.7 along with the programs. Figure 2.8 shows a moregeneral example of a task graph for fork-join parallelism. In general, a task graph is represented as a DAG in
which nodes represent tasks and edges represent task dependencies, but in the context of fork-join parallelism,
we use a diﬀerent expression following that of Blumofe and Leiserson [26]. In this expression, a “task” refers
to a series of computations that can be forked and joined, which is more like a “thread.” Instead, nodes in
DAGs are called strands, which are serial computations without any fork-join operations. In Figure 2.8, tasks
are enclosed by dashed lines, and tasks have multiple consecutive strands. Dependencies between strands are14

2. Background and Preliminaries

T1
T2

T2

T3

T1

T3

T2

T6

T4

T6

T3 T5

T7

T1
T8
W1

Steal

Steal

W3

W2

W4

Steal

T1
T2

T2

T1

T3

T3

T6

T4

W1

T6
T3

T7

W2

W3

W4

W1

W2

W3

W4

W1

W2

T2

T1

T5

T8

W3

W4

W1

W2

W3

W4

(a) W1 is executing tasks (b) W2 stole the oldest task (c) W3 and W4 stole tasks (d) Result of scheduling.
in the depth-ﬁrst order.
in W1. W1 popped afrom W1 and W2, respecClose tasks are likely to
Tasks (continuations) aretask from its local queue
tively.
be executed by the same
pushed to its local queue.
in LIFO order.
worker.

Figure 2.10.: Demonstration of task scheduling by work stealing (work-ﬁrst policy). Strands are colored by
the workers (W1, W2, W3, and W4) that executed them.
described by the three types of edges: fork (spawn) edges, continue edges, and join edges. A fork edge goesfrom the strand immediately preceding the fork call to the ﬁrst strand of the new task. A continue edge connect
consecutive strands within a task. A join edge goes from the last strand of a task to the strand immediately
following the corresponding join call in its parent task.
In fork-join parallelism, the parent-child relationships between active tasks (that have not been joined) form
a tree structure at any point of execution. Figure 2.9 is an example task tree, which corresponds to Figure 2.8
at a certain point in time. This ﬁgure suggests that task T3 has already been joined, and tasks T10 and T11
have not yet been forked.

2.2.2. Work Stealing
Work stealing [26] is arguably the most widely used load balancing algorithm today. We deﬁne workers as
virtualized processors, which are typically substantiated as processes or kernel-level threads. In work stealing,each worker maintains its own task queue, from which and to which the local worker pops and pushes tasks.
When the local queue is empty, the worker attempts to steal a task from another worker by selecting a victim
uniformly at random. Steal attempts are repeated by reselecting victims until a new task is successfully stolen.
Typically, the task queue is implemented as a double-ended queue (we assume that the THE protocol [9] isused); the worker pushes tasks to and pops tasks from the same end of the queue, whereas other workers stealtasks from the other end of the queue. Hence, local push/pop operations follow the last-in ﬁrst-out (LIFO)
order, and steal operations follow the ﬁrst-in ﬁrst-out (FIFO) order. Thus, the oldest task in the queue, whichis expected to have the largest amount of work, is stolen.
An important choice in implementing work stealing is child stealing or continuation stealing.
Child stealing A worker pushes a newly created child task to the local task queue and continues to run theparent task. A stealable task can be simply represented by a function pointer (or a closure) and its
arguments, as its execution has not started yet. Because of its simplicity, child stealing is commonlyused in library implementations, including oneTBB [11] and Microsoft PPL [13]. Child stealing is also
called help-ﬁrst [43] or parent-ﬁrst scheduling.

15

2. Background and Preliminaries
Continuation stealing A worker immediately executes the newly created child task and pushes the contin-

uation of the parent task to the local task queue. The continuation, which is the current blocked stateof the task execution including local variables, is stolen and resumed by another worker. Although it
is more complicated than child stealing, eﬃcient continuation stealing can be realized through special
language supports as in Cilk [8] or implemented by using user-level threads such as MassiveThreads [15].
Continuation stealing is also called work-ﬁrst or child-ﬁrst scheduling.
One beneﬁt of continuation stealing is that it preserves the serial execution order. The serial execution orderof fork-join parallelism is achieved if we sequentially execute tasks in the order as they appear in the program,ignoring all fork-join primitives. Tasks in Figure 2.8 are numbered in the serial execution order. This property
helps to prove good asymptotic bounds on execution time, space, communication [26], and data locality [27].
Blumofe and Leiserson [26] proved that the execution time of continuation stealing on 𝑃 processors isbounded by 𝑇𝑃 ≤ 𝑇1 /𝑃 + 𝑂 (𝑇∞ ), where 𝑇1 is the total amount of work (i.e., the serial execution time) and𝑇∞ is the critical path length (span) of the DAG (i.e., the execution time on an inﬁnite number of processors).
This bound is analogous to the greedy-scheduling theorem or Brent’s theorem [44] (𝑇𝑃 ≤ 𝑇1 /𝑃 + 𝑇∞ ) for an
ideal greedy scheduler. A scheduler is greedy if any pair of a ready task and an idle worker does not exist at
any point in time, i.e., an idle worker immediately executes a ready task if one exists, which is an important
property in general to keep processors as busy as possible. This bound suggests that if suﬃcient parallelism
is provided (i.e., 𝑇1 /𝑇∞ ≫ 𝑃), we have 𝑇𝑃 ≃ 𝑇1 /𝑃, which means linear speedup. 𝑇1 /𝑇∞ is referred to as
parallelism, which is a useful metric to estimate the number of cores that the computation can scale to.
Figure 2.10 illustrates an example schedule by continuation stealing. The execution begins with the root task
T1, and worker W1 executes tasks in the serial execution order (i.e., child-ﬁrst order), pushing continuations
into its local queue. The continuations that are the closest to the root are stolen ﬁrst. Other workers steal
continuations and resume their execution, following the serial execution order as well. As a result, many tasks
are forked and joined sequentially by the same worker. Tasks that are close to the root are likely to be stolen,
but other tasks, which are a majority, are unlikely to be stolen. This observation is supported by a result fromthe theoretical analysis [26]: the number of steals is 𝑂 (𝑃𝑇∞ ). If 𝑇1 /𝑇∞ ≫ 𝑃, the total number of tasks 𝑂 (𝑇1 )is much larger than the number of steals 𝑂 (𝑃𝑇∞ ).
The work-ﬁrst principle [9] is a implementation guideline to leverage this property of work stealing.
The work-ﬁrst principle suggests that the overheads that appear in ordinary fork-join operations should be
aggressively moved to those incurred by work-stealing events. This is because fork-join operations are
expected to be much more frequent than work-stealing events. In other words, fork-join operations should be
optimized for cases where no steal happens, without caring too much about overheads for work stealing.

2.2.3. User-Level Threading
In implementing fork-join task-parallel runtime systems, we can use OS-provided kernel-level threads astasks, but they are heavyweight and thus not suited for ﬁne-grained parallelism. Instead, user-level threads
are often used for this purpose, particularly in systems supporting continuation stealing and dynamic tasksuspension. User-level threads can be created, context switched, and destroyed, totally in user space without
the involvement of the kernel. Threading operations of user-level threads are extremely lightweight comparedwith kernel-level threads, because they can avoid heavyweight context switching to kernel space. In addition,scheduling of user-level threads can be fully controlled in user space. Due to these beneﬁts of lightweight andﬂexible threading, many user-level thread libraries have been developed: Qthreads [45], MassiveThreads [15],
Argobots [46], ComposableThreads [47] and so on.

16

2. Background and Preliminaries
The contexts (e.g., call stacks) of user-level threads are allocated at user space, and they are dynamically
mapped to and unmapped from the underlying kernel-level threads. Typically, as many kernel-level threads as
cores are created at program startup, over which many user-level threads are switched around. This threading
model is often referred to as the M:N threading model, where 𝑀 user-level threads are mapped to 𝑁 kernel-level
threads (typically 𝑀 ≫ 𝑁).
User-level threads can be dynamically suspended and resumed by user-level context switching. The implementation of context switching usually requires architecture-speciﬁc assembly code. As with kernel-level
threads, user-level threads have their own call stack region. When a thread is suspended, the running thread’s
state, including registers, is saved on its stack. When a thread is resumed, the thread state is restored fromthe information saved on its stack. In fact, the suspension capability varies across implementations, which iswell summarized in the work by Iwasaki et al. [48], [49]. User-level threads are typically cooperative (i.e.,
nonpreemptive), which means that threads are not evicted from cores unless threads voluntarily yields theircores. Exceptionally, preemptive user-level threads have been explored [2].
In task-parallel runtime systems based on user-level threads, tasks are simply represented as user-level
threads. Continuation stealing can be realized by suspending the current thread when forking a new thread,and resuming it either when popped from the local worker or when stolen by another worker. When joininga child thread, the parent thread is suspended if the child thread has not been completed yet, so that it can beresumed when the child thread is completed later. The join implementation will be discussed in Chapter 4
in more detail. In the following, user-level threads are simply referred to as “threads,” and we use the terms“tasks” and “threads” interchangeably in the context of fork-join parallelism.

2.2.4. Data Locality
In fork-join parallelism, sequential execution of tasks often exhibits a high degree of locality of reference. Thisis because tasks that are close in the task graph are likely to access the same or close memory regions. In a task
tree like Figure 2.9, tasks that share a close familial relationship are referred to as close tasks. For example, in
the recursive parallel merge sort code introduced in Figure 2.5, two recursive tasks for sorting compute on acontiguous region of memory, and after these two tasks are joined, the same memory region is accessed again
for merging. This is a typical situation in divide-and-conquer algorithms; as data are subdivided into smallerparts recursively, the ranges of data accessed by child tasks are getting smaller.
As demonstrated in Figure 2.10, in work stealing, close tasks are likely to be executed by the same worker
in the serial execution order. Therefore, in work stealing, each worker well exploits locality in fork-joinparallelism. The cache behaviour of work stealing has also been theoretically analyzed: Acer et al. [27]bounded the number of additional private cache misses in work stealing by 𝑂 (𝐶𝜏), where 𝐶 is a private cache
size for each worker and 𝜏 is the number of steals. As the number of steals is 𝑂 (𝑃𝑇∞ ), additional 𝑂 (𝐶𝑃𝑇∞ )cache misses are incurred compared to the number of cache misses that would be incurred in serial execution.
This shows that the number of private cache misses increases only in linear to the number of processors, whichsuggests high scalability of work stealing.
This bound means that only 𝑂 (𝐶) additional cache misses are incurred for each steal. In the following, we
provide informal but intuitive explanation for this bound. Suppose that a task at worker A is stolen by worker
B. As worker B initially caches data that are likely to be unrelated to that task, we consider that the initial cacheof worker B is empty. Therefore, in the early stages of executing the stolen task, we get additional cache missesthat would not have been incurred in the serial execution. However, once the cache is ﬁlled with newly fetcheddata, there comes a point where the cache state in worker B is identical to what it would have been if worker A

17

2. Background and Preliminaries
had continued executing the task sequentially without any steal. After that point, the number of cache misses
in worker B becomes identical to those that would have been incurred in worker A. Therefore, the number of
additional cache misses incurred by one steal is bounded by the number of cache misses required to ﬁll thecache of worker B, which is 𝑂 (𝐶). The same consideration is needed for computations after joining the stolentask, which also gives the same bound for each steal event.
However, work stealing is not optimal for hierarchical memory, while it works well for systems with only
private caches. The fact is that work stealing does not consider any machine hierarchy; it chooses a victim
for work stealing uniformly at random from all workers. Consequently, the mapping of hierarchical tasks infork-join parallelism does not match the memory hierarchy. In other words, tasks that are close in the task
graph are likely to be executed by workers that are far in the memory hierarchy. This reduces the chances
to eﬃciently utilize shared caches by sharing cached data among multiple workers. Furthermore, because of
random and nondeterministic scheduling by work stealing, it is challenging to match the task placement andthe data placement in NUMA architectures (Section 2.1.2) and distributed-memory computers. We addressthese locality issues by almost deterministic work stealing (ADWS) (Chapter 6) in this dissertation.
Another locality issue is that work stealing can cause an excessive number of capacity misses for sharedcaches. As demonstrated in Figure 2.10, workers tend to execute tasks that are far from the tasks that are
executed by any other worker. This is because they preferably steal tasks that are close to the root. This
property reduces the number of steals and private cache misses, but this is not a good property for sharedcaches. If the working set size for the entire computation does not ﬁt into a shared cache, executing allcomputations in parallel will cause many capacity misses. In such a situation, it is often required to execute
only a part of work that ﬁts into the shared cache (cf. cache blocking). However, work stealing does not careabout the shared cache capacity, and thus it is not optimal for shared caches. An experimental study [50]showed that work stealing underperformed shared-cache-aware parallel depth-ﬁrst (PDF) scheduling [51] on
systems with a single shared cache. Nevertheless, PDF scheduling is designed for systems with a single sharedcache, not for systems with private caches. This situation necessitates scheduling algorithms that work wellfor deep hierarchical caches, which have both private and shared caches. In this dissertation, we will introduce
multi-level scheduling (Section 6.4) to tackle this locality challenge.

2.2.5. Distributed Task-Parallel Systems
In this section, we introduce task-parallel runtime systems designed for distributed memory. We will coversystems that support dynamic task parallelism, but not limited to fork-join parallelism.
Here, we introduce an additional class of task parallelism for distributed memory. A system supports global
task parallelism if task dependencies are managed across diﬀerent compute nodes. In fork-join parallelism,
this means that tasks are dynamically scheduled across multiple compute nodes. For example, distributed
task-parallel systems such as Chapel [16], HabaneroUPC++ [52], and AsyncSHMEM [53] do not support
global task parallelism. In these systems, task-parallel execution is limited to within each node, disallowing
task parallelism spanning over multiple nodes.
Recent advances in network interconnects, especially RDMA (Section 2.1.3), have motivated researchers toinvestigate eﬃcient inter-node work stealing for global task parallelism [28], [29], [54]–[59]. However, manyof these implementations come with limitations. For example, some support fork-join parallelism but cannot
follow the work-ﬁrst policy (i.e., continuation stealing) [57]–[59], and others only support the bag-of-tasks
model, where tasks cannot be joined independently [54]–[56]. The bag-of-tasks model satisﬁes Conditions 1,
2, and 4 but not Condition 3 (Section 2.2.1), because all non-root tasks must be joined by the root task at the18

2. Background and Preliminaries
same time.
Excluding language-level approaches [14], [60], to the best of our knowledge, only the uni-address
scheme [28], [29] supports both fork-join parallelism and the work-ﬁrst policy on distributed memory at
the library level. The uni-address scheme spawns tasks as user-level threads and enables dynamic migrationof threads across diﬀerent nodes. This is achieved by dynamically copying call stacks of threads to othernodes while preserving their virtual addresses on diﬀerent processes. For the work-ﬁrst policy, its workstealing scheduler steals the continuation (call stacks) of threads in a fully one-sided (asynchronous) manner
by utilizing RDMA.

2.3. Global Address SpaceIn the previous section, we discussed dynamic load balancing for computations on both shared and distributed
memory. However, data are distributed over multiple compute nodes in distributed-memory computing. A
common practice is to exchange data between nodes by explicitly issuing two-sided or many-to-many communication via MPI (without RMA). However, this is not suited for asynchronous, dynamic task parallelism,
because the timings and locations of task execution are determined at runtime.
Instead, global address space has been investigated as a high-level shared-memory abstraction for distributed
memory. Global address space provides a view of shared memory over distributed memory spanning overmultiple nodes. Global memory locations can be uniquely addressed by global pointers. As global memory
can be accessed with the same interface from any nodes, this is suited for dynamic task parallelism.
In this dissertation, we categorize the interfaces for global address space into two models: Distributed
Shared Memory (DSM) (Section 2.3.1) and Partitioned Global Address Space (PGAS) (Section 2.3.2). While
DSM allows for transparent access to global memory using virtual memory mechanisms, PGAS explicitly
distinguishes global and local memory accesses at the API level.

2.3.1. Distributed Shared Memory (DSM)
DSM provides a virtual address space for global memory. The virtual address space can be transparently
accessed by CPU load/store instructions. This enables existing shared-memory programs to run on multiple
nodes without program modiﬁcation. Transparent access to global memory is realized by trapping memory
access to virtual pages by using memory protection mechanisms provided by the OS. Read memory access istrapped to fetch remote data to the pages in the local memory, after which the pages can be accessed without
memory protection faults. Write memory access is trapped to mark pages as modiﬁed, so that the modiﬁed
data are propagated to other nodes. Data updates should be consistently managed across multiple nodes,
and most DSM systems implement coherent caches for global memory accesses at the software level. Cachecoherence is managed at the software level by updating memory protection (e.g., removing access privilege
for invalidation) to trap memory accesses issued by programs.
DSM systems have been intensively studied in 1990s, and many DSM systems, such as TreadMarks [61],[62] and Midway [63], have been developed. However, research in DSM decreased due to its performanceissues, but recent advances in RDMA-capable interconnects led to proposals such as ArgoDSM [37] andMENPS [64]. These DSM systems employ relaxed the memory consistency models, simplifying cachecoherence protocols. Sequential consistency for data-race-free programs (SC-for-DRF) [65] is one of therelaxed memory consistency models, which ensures sequential consistency [66] if programs have no data race.
Data-race-freedom implies that the SWMR invariant (Section 2.1.1) is already satisﬁed at the program level.

192. Background and Preliminaries
This assumption signiﬁcantly simpliﬁes the cache coherence protocol because data race is a major source ofcomplexity.
Distributed Cilk [60] is a DSM implementation of Cilk. Its memory consistency model is DAG consistency [67], [68], which deﬁnes memory ordering rules for fork-join parallelism. DAG consistency can beviewed as SC-for-DRF in the context of fork-join parallelism. That is, DAG consistency assumes data-race-freeprograms, in which no conﬂicting access from concurrent tasks is allowed for any memory location. Informally, DAG consistency ensures that all memory operations are ordered according to the fork-join relationship(edges in Figure 2.8). For example, write operations performed by a strand must be visible to all strands that
follow that strand in the DAG.
Distributed Cilk is conceptually close to Itoyori, but Itoyori is implemented as a library with the PGAS
model, unlike Distributed Cilk. We chose the PGAS model instead of DSM for the sake of performance.
Although many eﬃcient techniques have been proposed, DSM cannot avoid large overheads for trapping
memory protection faults and handling false sharing, which are all needed to preserve transparency. In otherwords, in designing Itoyori, we traded oﬀ transparency for performance.

2.3.2. Partitioned Global Address Space (PGAS)The PGAS model explicitly distinguishes between global and local memory access, in contrast to DSM. To
date, many PGAS languages, such as Co-Array Fortran [69], Uniﬁed Parallel C (UPC) [70], XcalableMP [71],Chapel [16], and X10 [17], and PGAS libraries, such as OpenSHMEM [72], Global Arrays [73], UPC++ [74],
DASH [75], and HPX [76]1 have been developed. Some PGAS systems (e.g., X10, HPX) do not oﬀer a direct
way for accessing remote memory; instead, they encourage to move computations to the data owners (i.e.,
active messages). However, our focus is on PGAS systems that have direct read/write APIs to remote memory
without the use of active messages, because the task scheduler determines the mapping of computations in
Itoyori. The GET/PUT APIs are commonly used in PGAS systems to copy data between global and local
memory, and typical PGAS languages (e.g., UPC, Chapel) implicitly insert these GET/PUT APIs where global
objects are dereferenced. Although the details vary, the GET/PUT APIs typically appear as follows:• void GET(gptr t from ptr, void* to addr, size t size);• void PUT(void* from addr, gptr t to ptr, size t size);
They copy data of size bytes between the given local and global memory. The local memory has to bepre-allocated by the user. The representation of global pointers (of type gptr_t) varies across PGAS systems
and is not necessarily raw virtual addresses.
Commonly, the user can specify the memory distribution policy for global memory at the allocation time.
Popular memory distribution policies are block distribution, which distributes memory evenly among the nodes
so that each node’s memory is contiguous, and block-cyclic distribution, which distributes ﬁxed-size memory
chunks among nodes in a round-robin fashion. Most PGAS systems have a way to directly access the local
portions of global memory determined by the memory distribution policy (e.g., by converting global pointers
to local raw pointers). This helps users to follow the owner-computes rule [77] (i.e., the data owner node
should compute on the local data) for better performance.
Arguably, the owner-computes rule assumes SPMD and is diﬃcult to apply to irregular parallelism. Nevertheless, systems including Scioto [54], HotSLAW [58], Grappa [59], and extensions to X10 [78], [79] support
1 HPX refers to its global address space as Active Global Address Space (AGAS) to diﬀerentiate it from PGAS in that AGAS allows

transparent relocation of global memory. However, in this dissertation, we view it within a broader scope of PGAS.

20

2. Background and Preliminaries
inter-node work stealing to handle irregular parallelism under the PGAS model. However, for the reasons mentioned in Chapter 1, they often incur ﬁne-grained and redundant communication for ﬁne-grained parallelism.
This situation motivated us to investigate software caching techniques for PGAS systems (Chapter 5).

21

3. Programming with the Itoyori Runtime System
In this chapter, we explain the programming model and APIs of Itoyori. The content of this chapter is partly
based on our published paper [5], but more details of Itoyori APIs are discussed in this chapter.

3.1. Overview
Itoyori is implemented as a header-only C++17 library. Its communication is internally implemented with MPI,
particularly by using MPI-3 RMA (assuming the MPI WIN UNIFIED model). An Itoyori program is launched
as an MPI program (typically by invoking the mpiexec command), spawning multiple processes over multiple
compute nodes. A process typically corresponds to a CPU core, as Itoyori does not manage multiple kernel-level
threads within each process (eliminating the need for the MPI THREAD MULTIPLE support in MPI). Therefore,
multiple processes can exist within a single node, but Itoyori does not expose any APIs for distinguishing
between inter- and intra-node processes and between shared and distributed memory (for processor-oblivious
programming). Instead, the hardware topology is considered internally in its implementation to better utilize
hardware resources.
Itoyori is built on top of the concepts of global task parallelism and global address space. Global task
parallelism is supported by distributed multithreading based on the uni-address scheme, and global address
space is enabled in the form of PGAS. They enable extremely simple and ﬂexible distributed-memory programming: programmers expose logical parallelism as user-level threads and explicitly issue memory accesses
to a single, global address space. The programming interfaces for these two components, the multithreading
layer (Section 3.2) and the PGAS layer (Section 3.3), are designed to be loosely coupled for independent useof each component. However, direct use of these low-level interfaces is generally not recommended; instead,C++ high-level interfaces (Section 3.4), which are built on top of these two components, are intended to be
directly accessed by programmers. While these APIs are high-level and straightforward to use, they have some
pitfalls and limitations that programmers need to be aware of (Section 3.5).
In the following, we introduce Itoyori APIs with some C++ code examples. Itoyori APIs are preﬁxed with theityr:: namespace (short for “itoyori”) in C++. Speciﬁcally, the low-level multithreading layer is preﬁxed with
the ityr::ito:: namespace, and the low-level PGAS layer is with the ityr::ori:: namespace. Regularprogrammers are generally not recommended to directly access the low-level APIs in the ito and ori layers.
Primary high-level APIs are preﬁxed with a single ityr:: namespace.

3.2. Global Task Parallelism
In this section, we ﬁrst explain the parallel control ﬂow management in Itoyori’s ito layer. As Itoyori is built
on top of MPI, programs ﬁrst need to switch from MPI’s Single Program Multiple Data (SPMD) context to
Itoyori’s fork-join task-parallel context (Section 3.2.1). After that, the program can arbitrarily spawn user-level
threads for fork-join parallelism (Section 3.2.2).

22

3. Programming with the Itoyori Runtime System
12
3
4

int main() {ityr::init(); // Itoyori runtime initialization/* SPMD scope */auto my_rank = ityr::my_rank(); // Process ID starting from 0 (same as the MPI rank)
auto n_ranks = ityr::n_ranks(); // The total number of processes
printf("MPI Process %d/%d\n", my_rank, n_ranks);

5
6
7
8
9
10

ityr::root_exec([=] { // Spawn only one root task globally/* Fork-join scope */

11
12
13
14
15
16
17
18
19
20
21

ityr::coll_exec([=] { // Run a computation collectively on all processes/* SPMD scope */}); // returns when all processes exit from the inner SPMD scope/* Fork-join scope */}); // returns when the root thread is completed/* SPMD scope */ityr::fini(); // Itoyori runtime finalization}

Figure 3.1.: Itoyori APIs for switching between the SPMD and fork-join scopes.

3.2.1. Switching between SPMD and Task-Parallel Scopes
As Itoyori programs are launched as MPI programs, their execution begins with the SPMD model, in
which the same program is simultaneously executed by multiple processes. In order to begin fork-join parallel
execution, programmers need to create the root task for fork-join, while stopping the SPMD execution (becauseof Condition 1 in Section 2.2.1). Figure 3.1 shows a program example to switch between the SPMD and forkjoin execution1 . The ityr::root exec() API spawns the root thread, which is then executed by one ofthe processes. It returns when the root thread is completed, then resuming the SPMD execution again. A
program scope within ityr::root exec() is called a fork-join scope, while a scope outside it is an SPMD
scope. User-level threads can only be created within the fork-join scope. Conversely, the ityr::coll exec()
API can be used to switch from the fork-join scope to the SPMD scope. This is useful when dynamically
performing some operations correctively on all processes (e.g., collective memory allocation in Section 3.3.1).
This design makes it easy to integrate Itoyori with existing MPI applications, because programmers can freely
call any MPI calls within the SPMD scope.
Note that Itoyori requires each process have the same layout of virtual address space (i.e., no address spacelayout randomization) mainly for the uni-address scheme. This can be achieved by launching each process by
the setarch --addr-no-randomize command in Linux. Therefore, global variables in C/C++ (not in thecontext of PGAS) are located at the same virtual addresses for each process, and thus Itoyori allows for directaccess to global variables. This is particularly useful for managing constant parameters. However, it is on theuser’s responsibility to ensure consistency of values for global variables, as their values can be diﬀerently set
across diﬀerent processes. It is generally recommended to set the same value to a global variable across allprocesses in the SPMD scope, without modifying it in the fork-join scope.
1 While the core implementations for ityr::root exec() and ityr::coll exec() are in the low-level threading layer (e.g.,

as ityr::ito::root exec()), this code example shows high-level interfaces (preﬁxed with ityr::) with memory fences(Section 3.3.3) inserted properly.

23

3. Programming with the Itoyori Runtime System
1
2
3
4
5
6
7
8
9
10

long fib(int n) {if (n <= 1) {return 1;} else {ityr::ito::thread th([=] { return fib(n - 1); });long y = fib(n - 2);long x = th.join();return x + y;}}

Figure 3.2.: Fibonacci example with Itoyori’s low-level multithreading APIs.

3.2.2. Fork/Join APIs
In the fork-join scope, Itoyori can dynamically spawn user-level threads by using low-level threading
primitives (like Figure 2.4a in Section 2.2.1). Figure 3.2 calculates the Fibonacci sequence (𝐹 (𝑛) = 𝐹 (𝑛 −1) + 𝐹 (𝑛 − 2)) in a divide-and-conquer manner, by recursively spawning threads. Note that this is an extremely
ineﬃcient example only for demonstrating the API usage. We suppose that the fib() function is invokedwithin the fork-join scope. The ityr::ito::thread primitive can be used for forking a new thread andreturning its computed value when joining it (the join() method). In fact, multithreading in Itoyori is not
limited to fork-join parallelism, but the threads can be used as futures (see the LCS problem in Section 4.5.4 forthe usage). This means that threads can be joined by non-parent threads. However, Itoyori currently does not
expose futures as high-level APIs (preﬁxed with ityr::); high-level interfaces are designed only for fork-joinparallelism (e.g., ityr::parallel invoke()) for now.
One restriction in Itoyori’s multithreading is that access to local variables in other threads’ call stacks is
prohibited. That is, any pointers or references to local variables shall not be passed to other threads. This is
why lambda expressions in Itoyori programs capture values by copy. In addition, access to process-local heap
memory is not guaranteed across fork/join calls, as threads can be dynamically migrated to other processes
at fork/join calls. This means that, for example, access to the standard std::vector is not allowed acrossfork/join calls, because it allocates heap memory, and Itoyori does not migrate heap-allocated memory alongwith threads. Note that access to local variables in the same thread’s stack is possible across fork/join calls,
thanks to the uni-address scheme (see Chapter 4 for more details). Hence, in Itoyori’s PGAS model, local
memory is deﬁned in terms of each thread stack, rather than physical memory local to each process. As accessto local memory in other threads is not allowed, in order to share data among multiple threads, users need toeither pass variables by value at forking or exchange data via global memory.

3.3. Global Address Space
In Itoyori, global memory objects are allocated/deallocated from the global heap by specifying the memory
distribution policy (Section 3.3.1). The returned global addresses are merely raw, often 64-bit, virtual addresses.
However, a program cannot directly access the virtual addresses unless checkout/checkin calls (Section 3.3.2)
are made for the accessing region. Global memory accesses with checkout/checkin APIs must be performed
in a data-race-free manner, and memory fences (release/acquire APIs in Section 3.3.3) ensure the memoryordering across diﬀerent processes.

24

3. Programming with the Itoyori Runtime System
12
3
4/* SPMD scope */// Allocate 10000 bytes of memory collectively from multiple processes in the SPMD scope
void* gptr_coll = ityr::ori::malloc_coll(10000);

5
6
7
8
9
10

ityr::root_exec([=] {/* Fork-join scope */// Allocate 10 bytes of memory from the local process asynchronously to other processes
void* gptr_noncoll = ityr::ori::malloc(10);

11
12
13
14
15
16
17

/* ... (the thread can be dynamically migrated to other processes) */// Free non-collective memory asynchronously to other processes
ityr::ori::free(gptr_noncoll, 10); // The memory size is needed for non-collective free});// Free collective memory in the SPMD scope
ityr::ori::free_coll(gptr_coll);

Figure 3.3.: Low-level memory allocation APIs in Itoyori.

3.3.1. Memory Allocation and Distribution Policies
Itoyori extends the common malloc() interface so that the user can specify a memory distribution policy.
The policy is either one of the collective policies or the non-collective policy. Figure 3.3 shows a program
example for allocating both collective and non-collective memory.
Collective distribution policies are used to allocate a large amount of memory that spans over multiple
nodes. Itoyori currently supports the block and block-cyclic distribution policies (see Section 2.3.2) ascollective distribution policies. For collective memory, the allocation function (ityr::ori::malloc coll())
and deallocation function (ityr::ori::free coll()) must be called collectively by all processes in the
SPMD scope. In order to perform collective memory allocation in the fork-join scope, users need to callityr::coll exec() to switch to the SPMD scope and then call ityr::ori::malloc coll(). The allocation
call ityr::ori::malloc coll() can accept an additional argument to specify the memory distribution policy(e.g., block or block-cyclic).
In contrast, the non-collective policy allows eﬃcient ﬁne-grained memory allocation asynchronous to otherprocesses, both in the SPMD scope and in any threads in the fork-join scope. With the non-collective policy,
memory objects are allocated only from the local process without the involvement of any other process.
Nevertheless, the allocated memory can be remotely accessed with checkout/checkin APIs, and even freed byany process. Allocation and deallocation of non-collective memory are performed in ityr::ori::malloc()
and ityr::ori::free(). Note that ityr::ori::free() requires the memory size in addition to the global
pointer to be freed. This is because of implementation eﬃciency, but this is not a severe limitation because
this detail is hidden by high-level containers (Section 3.4.2)
A global pointer is returned at memory allocation, which can be used for accessing global memory via
checkout/checkin APIs (Section 3.3.2). As a global pointer is merely a single virtual address, it can bearbitrarily passed to any threads by value. In addition, Itoyori also provides the global ptr class, which canbe used for representing a global pointer but prevent direct dereferencing without calling checkout/checkin
APIs.

25

3. Programming with the Itoyori Runtime System
1
2
3
4
5
6
7
8
9
10
11
12
13

template <typename T>
void mergesort(T* a, size_t n) {if (n < cutoff) {ityr::ori::checkout(a, n, ityr::ori::checkout_mode::read_write);
std::sort(a, a + n);
ityr::ori::checkin(a, n, ityr::ori::checkout_mode::read_write);} else {ityr::parallel_invoke([=] { mergesort(a
, n/2
); },
[=] { mergesort(a + n/2, n - n/2); });merge(a, n/2, n);}}Figure 3.4.: Recursive parallel merge sort implemented in Itoyori using low-level checkout/checkin APIs.

3.3.2. Checkout/Checkin APIs for Global Memory Access
Checkout/Checkin APIs are used to gain access to a global memory region. Figure 3.4 shows a recursive parallel merge sort example written in Itoyori by using low-level checkout/checkin APIs. Only the diﬀerence from
Figure 2.5 is the insertion of ityr::ori::checkout() and ityr::ori::checkin() calls. Checkout/CheckinAPIs must be called in pairs, and each pair requires the exact same arguments. While a global memory regionis checked out by the local process, it can directly access the region by using the same virtual addresses withmemory load/store instructions. Checkout/Checkin APIs are deﬁned as follows:
void checkout(void* addr, size t size, checkout mode mode); claims that the program will accessthe memory of the half-open region [addr, addr + size) in the speciﬁed access mode. The requested
memory region becomes accessible until checkin() is called for that region. The checkout mode can beeither read, read write, or write. If mode is read or read write, the system considers a read event
for [addr, addr + size) happens at this point and may fetch the latest data from remote nodes. If the modeis write (write-only access), the region may be left uninitialized.
void checkin(void* addr, size t size, checkout mode mode); claims that access to the previously
checked-out memory is completed. The arguments for addr, size, and mode must be exactly the sameas those passed to the previous, corresponding checkout call. This checkin function should be called
once and only once for each checkout call as a pair. If the mode is read write or write, the system
considers a write event for [addr, addr + size) happens at this point, and this entire region is considered
modiﬁed.
Note that in the read write or write mode, all bytes of the checked-out data are considered dirty, even if
the program did not actually update the data. In other words, the access mode in Itoyori is not like an accessprivilege, but more like memory load/store operations. Thus, for example, always specifying the read write
mode is not a conservative approach; it is a data race if diﬀerent processes concurrently checks out the same
region in the read write mode, even if they do not actually write to the region.
As long as the program is data-race-free, multiple processes can simultaneously check out the same region.
In other words, multiple processes can check out the same region only in the read mode; otherwise only one
process can check out the region at the same time. Within each process, multiple checkout requests can be26

3. Programming with the Itoyori Runtime Systemsimultaneously made for the same region in any access mode, but they must be checked in before program
points where threads can migrate (e.g., fork/join calls).
Because the (user-conﬁgurable) cache size is ﬁxed in Itoyori, the amount of memory that can be simultaneously checked out by each process is limited. If it exceeds the cache size limit, a checkout function throws an
exception. For example, if a process sweeps over a large global array that does not ﬁt into the cache, it cannot
check out the entire array at once. Instead, it has to break checkout/checkin requests into suﬃciently small
chunks and process each chunk in turn. While this may seem cumbersome, the details can be abstracted away
by using high-level patterns for range-based operations (Section 3.4.3). By using these high-level patterns, thesystem can automatically determine proper chunk sizes. This design allows us to easily handle huge data that
do not ﬁt into a single node.
In fact, Itoyori also supports conventional GET/PUT APIs (Section 2.3.2). Why we have introduced novel
checkout/checkin APIs is explained in Section 5.2.

3.3.3. Release/Acquire APIs for Synchronization
Updates to global memory should be propagated to other processes to ensure a consistent global view ofmemory. Itoyori’s memory consistency model is sequential consistency for data-race-free programs (SCfor-DRF) [65], which is also used in many languages such as C/C++11, Java, UPC [70], and Chapel [16].
SC-for-DRF is a relaxed memory model that ensures well-deﬁned memory ordering (sequential consistency)
as long as a program has no data race. Hence, in Itoyori, global memory access (i.e., checkout/checkin calls)
should be performed in a data-race-free manner. Under this memory model, cache coherence is properly
managed by the runtime system.
Under this relaxed memory model, a simple cache coherence protocol can be used, assuming that thedata-race-freedom is already ensured by programmers. Following the convention (e.g., [80]), Itoyori oﬀers
release and acquire memory fences to ensure a consistent global view of memory. Informally, if a releasefence happened before an acquire fence, there is a synchronization order, i.e., all updates made before therelease fence must be observed after the acquire fence. These fences are typically hidden from the user andencapsulated by synchronization primitives, such as locks, barriers, and fork-join calls. As Itoyori currently
supports only fork-join, the memory model is equivalent to DAG consistency [67], [68].
Figure 3.5 shows an example program to demonstrate the usage of release/acquire fences. This code
concurrently updates the two global memory locations x and y by spawning a new thread. To properly
propagate these changes, release/acquire fences must be inserted to where inter-process synchronization canhappen (i.e., fork/join operations). Because both the child and parent threads can be migrated to other processes
at fork/join calls, release/acquire fences (ityr::ori::release() and ityr::ori::acquire()) are inserted
accordingly. Section 5.4.1 discusses how to insert memory fences to fork-join primitives in more detail.
These memory fences are usually not visible to regular programmers and encapsulated by high-level fork-joinprimitives such as ityr::parallel invoke() and parallel loops (Section 3.4.3). These implementations may
omit some memory fences for eﬃciency. Section 5.4.2 discusses how to reduce the number of memory fencesin the work-stealing scheduler in Itoyori.

3.4. High-Level C++ APIs
Thus far, we have introduced basic, low-level APIs for multithreading and PGAS in Itoyori. In this section,
we explain higher-level APIs built on top of them, which align well with modern C++ programming. We ﬁrst

27

3. Programming with the Itoyori Runtime System1

int *x = /* ... */, *y = /* ... */; // global pointers

2
3
4
5
6
7

/* Initialize x and y */ityr::ori::checkout(x, 1, ityr::checkout_mode::write);
ityr::ori::checkout(y, 1, ityr::checkout_mode::write);*x = 1; // x <- 1
*y = 1; // y <- 1
8 ityr::ori::checkin(x, 1, ityr::checkout_mode::write);
9 ityr::ori::checkin(y, 1, ityr::checkout_mode::write);
10
11
12
13

/* Spawn a thread */ityr::ori::release();ityr::ito::thread th([=] {14
ityr::ori::acquire();
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

/* Update x */ityr::ori::checkout(x, 1, ityr::checkout_mode::read_write);*x += 1; // x <- 1 + 1
ityr::ori::checkin(x, 1, ityr::checkout_mode::read_write);
ityr::ori::release();});
ityr::ori::acquire();/* Update y */ityr::ori::checkout(y, 1, ityr::checkout_mode::read_write);*y += 1; // y <- 1 + 1
ityr::ori::checkin(y, 1, ityr::checkout_mode::read_write);/* join */ityr::ori::release();th.join();ityr::ori::acquire();/* Assert x and y */ityr::ori::checkout(x, 1, ityr::checkout_mode::read);
ityr::ori::checkout(y, 1, ityr::checkout_mode::read);assert(*x == 2); // x -> 2
assert(*y == 2); // y -> 2
ityr::ori::checkin(x, 1, ityr::checkout_mode::read);
ityr::ori::checkin(y, 1, ityr::checkout_mode::read);

Figure 3.5.: Usage of low-level release/acquire APIs with fork/join APIs and checkout/checkin APIs.

28

3. Programming with the Itoyori Runtime System
1
2
3
4
5
6
7
8
9
10
11
12
13
14

template <typename T>
void mergesort(ityr::global_ptr<T> a, size_t n) {if (n < cutoff) {ityr::checkout_span<T> a_cs =ityr::make_checkout(a, n, ityr::checkout_mode::read_write);
std::sort(a_cs.begin(), a_cs.end());// The checkin API is automatically called when the checkout span is destroyed} else {ityr::parallel_invoke([=] { mergesort(a
, n/2
); },
[=] { mergesort(a + n/2, n - n/2); });merge(a, n/2, n);}}Figure 3.6.: Recursive parallel merge sort implemented in Itoyori using the checkout span.
introduce checkout spans (Section 3.4.1), which safely manage checked-out memory. Then, we discuss how
to deal with C++ containers (Section 3.4.2) and parallel loops (Section 3.4.3) in Itoyori.

3.4.1. Checkout SpansAs explained in Section 3.3.2, a pair of the checkout and checkin APIs require exactly the same arguments.
In addition, a checked-out memory region must be checked in exactly once. To enforce these requirements,
Itoyori provides the ityr::checkout span container. This supports operations similar to those of the C++20
std::span container, but Itoyori’s checkout span does not allow copy operations. Therefore, similar to uniquepointers (std::unique ptr), checkout spans ensure that checked-out memory is checked in exactly once bycalling the checkin API when the destructor is called.
Figure 3.6 shows the usage of checkout spans in the example of recursive parallel merge sort. The
make checkout() receives the arguments to perform the checkout operation and returns a checkout span. A
checkout span is composed of a raw pointer and a range size (as in std::span), allowing for direct access tothe checked-out memory. When the checkout span is destroyed, this region is checked in. In addition, it can
explicitly perform the checkin operation by calling the checkin() method.

3.4.2. Global Containers
In C++ programming, containers are often used instead of directly allocating memory (e.g., malloc/freecalls) for convenience and safety. Itoyori also provides global containers to manage global memory insidecontainers. Programmers are generally recommended to use global containers, rather than directly calling
low-level memory allocation APIs (Section 3.3.1).
A global vector (ityr::global vector) is a representative global container, which manages a contiguous region of global memory. In addition to the operations in the standard std::vector, Itoyori’s global
vector accepts extra options (ityr::global vector options). The global vector options include whether
to allocate collective memory or not, whether to parallelize construction/destruction for vector elements, andthe cutoﬀ size for internal parallelization. Figure 3.7 shows an example usage of global vectors. The option
ityr::global vector options(true) speciﬁes that it must allocate collective memory; therefore the memory allocation must be performed collectively by all processes. If memory allocation or deallocation happens29

3. Programming with the Itoyori Runtime System
12
3
4// Allocate collective memory with a global vector in the SPMD scope
ityr::global_vector<int> v1(ityr::global_vector_options(true), 100);// Create a global span that covers the global vector v1
ityr::global_span<int> s1(v1);

5
6
7
8
9
10

ityr::root_exec([=] {// Update collective memory of v1 by checking out global span s1.
// Here, v1 should not be passed to the lambda expression to prevent copy.
{
auto s1_cs = ityr::make_checkout(s1, ityr::checkout_mode::read_write);11for (int& x : s1_cs) x += 1;
12
}
13
14
15
16
17
18
19
20
21

// Allocate collective memory with a global vector in the fork-join scope
ityr::global_vector<int> v2(ityr::global_vector_options(true), 200);// Allocate non-collective memory with a global vector
ityr::global_vector<int> v3(10);/* ... */});Figure 3.7.: Global vector and span containers in Itoyori.
ityr::global_span<int> s1(/* ... */);/* ... */ityr::transform(
ityr::execution::par,// Parallel execution policy
s1.begin(), s1.end(),// The input range (pair of the begin and end iterator)
s1.begin(),// The output range (beginning from the begin iterator)
7[](int x) { return x + 1; }); // Add 1 to each element
1
2
3
4
5
6

Figure 3.8.: High-level transform call in Itoyori (corresponding to a loop in Figure 3.7).
in the fork-join scope, it internally calls ityr::coll exec() to perform collective operations in the SPMD
scope. This enables global vectors to be initialized both in the SPMD scope and in the fork-join scope.
One important note is that in Itoyori’s programming style, global vectors can easily be copied unintentionally.
For example, in Figure 3.7, global vector v1 is allocated in the SPMD scope and its elements are accessed in thefork-join scope. However, as threads cannot take references to outside the threads, the vector would be required
to be copied across ityr::root exec(), as explained in Section 3.2.2. This would result in copying theentire elements in the vector, incurring substantial overheads. Instead, one can extract global pointers from theglobal vector and copy global pointers across lambda expressions, but this approach cannot take advantage ofhigh-level containers. Hence, Itoyori provides a global span (ityr::global span) that resembles the standard
C++20 std::span(). A global span is essentially a pair of a global address and a range size, and it does nothave any ownership for the memory range. Therefore, it is trivially copyable across ityr::root exec() orfork/join calls, enabling high-level array range management in Itoyori. The ityr::make checkout() call can
accept global spans as its ﬁrst argument instead of a global pointer and a checkout size.

3.4.3. Parallel Loops and Global Iterators30

3. Programming with the Itoyori Runtime Systemityr::global_span<int> s1(/* ... */);/* ... */ityr::for_each(
ityr::execution::par,ityr::make_global_iterator(s1.begin(), ityr::checkout_mode::read_write),
ityr::make_global_iterator(s1.end() , ityr::checkout_mode::read_write),7[](int& x) { x += 1; }); // Add 1 to each element
1
2
3
4
5
6

Figure 3.9.: Itoyori’s for each() call and global iterators (equivalent to Figure 3.8).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

void axpy(double a, ityr::global_span<double> xs, ityr::global_span<double> ys) {ityr::for_each(
ityr::execution::par,ityr::make_global_iterator(xs.begin(), ityr::checkout_mode::read),
ityr::make_global_iterator(xs.end() , ityr::checkout_mode::read),
ityr::make_global_iterator(ys.begin(), ityr::checkout_mode::read_write),[=](const double& x, double& y) {y += a * x;});}
double norm(ityr::global_span<double> xs) {double s2 =ityr::transform_reduce(ityr::execution::par,xs.begin(), xs.end(),ityr::reducer::plus<double>{},[](double x) { return x * x; });return std::sqrt(s2);}
int main() {ityr::init();ityr::root_exec([=] {double a = 0.1;
size_t n = 10000;
ityr::global_vector<double> x_vec(ityr::global_vector_options(true), n, 1.0);
ityr::global_vector<double> y_vec(ityr::global_vector_options(true), n, 1.0);
ityr::global_span<double> x(x_vec);
ityr::global_span<double> y(y_vec);// x = {1.0, 1.0, ..., 1.0}, y = {1.0, 1.0, ..., 1.0}
axpy(a, x, y);// x = {1.0, 1.0, ..., 1.0}, y = {1.1, 1.1, ..., 1.1}
std::cout << norm(y) << std::endl; // output = 110 (= sqrt(1.1 * 1.1 * 10000))});
ityr::fini();}Figure 3.10.: AXPY and L2 norm calculation using high-level parallel loops and global iterators.

31

3. Programming with the Itoyori Runtime System
In the previous example of Figure 3.7, the entire elements in the global vector are checked out at once, andthen they are iterated over in a for loop. However, as explained in Section 3.3.2, the entire vector might not ﬁt
into the software cache in each process if the vector is very large. In addition, a convenient way to parallelize
a for loop would be desired for convenience.
To that end, Itoyori provides high-level parallel loops based on iterator ranges, which resemble the C++17
parallel STL. Figure 3.8 shows an example of the transform operation in Itoyori. This operations is often
called a map operation, but it is called a transform operation in the context of C++ (corresponding to thestandard std::transform()). Its ﬁrst parameter is the execution policy (parallel execution in this case), andits second and third parameters constitute a input range of iterators. Each element in the input range is applied
the operator given as the ﬁfth parameter, and its result is written to the output range beginning from the iteratorof the fourth parameter. Since the same ranges are speciﬁed for the input and output range in this case, this is
an inplace update operation, which corresponds to the loop in Figure 3.7. By using high-level loop constructs
like this, when global pointers are provided as the input/output ranges, they can be automatically checked out
within these calls. In the transform operation, checkout modes can be inferred from its speciﬁcation (read for
input and read write or write for output2 ). Therefore, users do not have to explicitly call checkout/checkinAPIs or fork/join APIs.
Nevertheless, in some cases, checkout modes for given ranges are not clear from the speciﬁcation.
ityr::for each() (corresponding to the standard std::for each() is a basic loop construct, which takes
as iterator ranges and an operator for each element. However, there is no restriction for the operator; it may
or may not update each element. Therefore, users should provide the checkout mode for such cases. Global
iterators are designed for this purpose, which annotate global pointers with their checkout mode that should
be used for internal checkout/checkin calls in parallel loops. Figure 3.9 shows how to use global iterators with
the ityr::for each() construct, which is equivalent to the previous transform example (Figure 3.8).
In addition, ityr::for each() can accept multiple iterator ranges as arguments, unlike std::for each().
This enables more general loop-based algorithms to be expressed in this form. Figure 3.10 shows a program
example for AXPY. AXPY computes a scalar-vector product and adds the result to another vector: 𝑦 ← 𝑎×𝑥+𝑦,
where 𝑎 is a scalar and 𝑥 and 𝑦 are vectors. The axpy() function in the code does this calculation. In
addition, Figure 3.10 shows the code for L2 norm calculation. This is a reduction operation, and thus
ityr::transform reduce() is called with a reducer. As well as the transform operation, programmers do
not have to provide the checkout mode with global iterators because it is always the read mode.
These parallel loop constructs are internally parallelized in a two-way divide-and-conquer approach byusing fork/join primitives. Programmers can conﬁgure the cutoﬀ count for parallel execution by passing extra
parameters to the ityr::execution::parallel policy class.
Combining these high-level constructs in Itoyori, simple and concise parallel programs can be written, as
demonstrated in Figure 3.10. Even though this is a very simple example with regular parallelism, one beneﬁt
in using Itoyori is that it comes with automatic load balancing. That is, even if the computational amountof work for each iteration diﬀers across iterations in parallel loops, Itoyori automatically load-balances themacross diﬀerent compute nodes.

2 For the output ranges, the write checkout mode can be used if the element type is trivially copyable. Otherwise, read write should

be used because it can use its current state in its copy assignment operator.

32

3. Programming with the Itoyori Runtime System

3.5. Pitfalls and Current Limitations
As explained above, Itoyori enables high-level programming for both shared- and distributed-memory machines. However, there are certain pitfalls that hinder the proper use of Itoyori APIs. As a conclusion to thischapter, we will enumerate and review the unique pitfalls in programming with Itoyori. Then, we discuss thecurrent limitations and potential future solutions to these pitfalls.
In Itoyori, the following situations are considered pitfalls, which can lead to undeﬁned behaviours:
1. A global variable (in the context of C/C++) has diﬀerent values in diﬀerent processes (Section 3.2.1). If
the global variable is accessed in the fork-join scope, the result may become nondeterministic. In orderto read the same value from a global variable in any processes, programmers need to collectively set thesame value to it in the SPMD scope. In addition, there is no way to prevent programmers from updating
global variables in the fork-join scope.
2. A thread accesses local variables in other thread stacks (Section 3.2.2). This is prohibited in Itoyori
because the uni-address scheme (Chapter 4) does not provide access to call stacks of threads other thanthe current thread. As pointers or references to local variables can be easily passed to child threads
through the fork API, the user should specify copy captures to lambda expressions by default. Still, thiscan cause another issue of unintentional copy for global containers (Section 3.4.2).
3. A thread keeps pointers to the process-local heap memory across fork/join calls (Section 3.2.2). As
threads can be migrated to other processes at any fork/join calls, allocated process-local heap memory
must be freed before fork/join calls. The usage of STL containers such as std::vector can easily
cause this situation. To avoid this, the user can replace them with Itoyori’s global containers (e.g.,
ityr::global vector).
4. A global memory region is being checked out across fork/join calls (Section 3.3.2). All checked-out
memory must be checked in before any fork/join calls. Note that checkout spans do not automatically
perform checkin before fork/join calls because they are unaware of fork/join calls in the current implementation. This can be particularly problematic in the nesting of high-level parallel loops, in which
global iterators are automatically checked out inside parallel loops.
5. The program is not data-race-free (Section 3.3.2). If a data race happens to a global memory location,
the memory ordering for this location becomes undeﬁned. Even if the program has no actual races, theentire region checked out with the write access mode is considered modiﬁed in Itoyori, which can leadto the same situation.
Unfortunately, it is challenging to prevent or detect pitfalls 1, 2, and 3, as these behaviours are unmanageablefrom library implementations. In order to handle them, language-level approaches (e.g., designing a new
language or extensions to C/C++) would be required in the future. For pitfall 4, we consider that it is possible
to implement a runtime checking mechanism. For pitfall 5, we will be able to dynamically detect data races byapplying existing techniques on on-the-ﬂy data race detection for fork-join programs (e.g., [81]). In addition, it
may be possible to guarantee the absence of data races by statically performing ownership-based type checking
as seen in Rust.

33

4. Scalable Continuation Stealing on Distributed
MemoryThis chapter describes the implementation of distributed multithreading in Itoyori, which is based on theuni-address scheme scheme developed by Akiyama and Taura [28], [29]. The content of this chapter is about
experimental analysis and extensions to the uni-address scheme, which is based on our published paper [4].

4.1. Overview
Load balancing is crucial to performance in parallel computing, especially for algorithms that yield irregularparallelism. As manual load balancing is cumbersome and error-prone, runtime-level support for dynamic
load balancing is substantially important. Many parallel runtime systems that eﬃciently support dynamic loadbalancing have been developed and used in production [7]–[9], [12], [15], [45], [46], [82].
Work stealing [26] is arguably the most widely used load balancing algorithm for irregular parallelism: eachprocessor has a local task queue, from which other idle processors try to steal tasks by randomly selecting a“victim.” An important scheduling choice in work stealing is between child stealing or continuation stealing.
In child stealing, a processor places a newly created child task in the local task queue to potentially be stolenby another processor and continues to run the parent task. In continuation stealing, the processor immediately
executes the newly created child task and leaves the continuation of the parent task in the queue. Continuation
stealing preserves the serial execution order as much as possible, because a task is executed as an ordinary
function call unless its parent task is stolen. This helps to eﬃciently resolve task synchronization in fork-join
programs (“join” primitives) and prove theoretical bounds on execution time, space, communication [26], anddata locality [27], as discussed in Section 2.2.2. Because of these beneﬁcial characteristics, many sharedmemory runtime systems, including Cilk [8], [9], use continuation stealing.
On distributed memory, most of the existing work-stealing implementations are child stealing, and many
child-stealing implementations have shown good scalability with the use of RDMA [55], [56], [58]. The
introduction of RDMA has eliminated the need for frequent interruptions to the victim processors, which areotherwise needed for steal attempts with two-sided (message-based) communication [78], [83]. Nevertheless,continuation stealing is still rare on distributed memory. This is presumably due to two assumptions:
1. Implementing continuation stealing as a library with native (unmodiﬁed) C/C++ compilers is diﬃcult
because it involves migration of call stacks across nodes.
2. Continuation stealing is less eﬃcient than child stealing because a call stack is typically larger than thefunction pointer and its arguments needed for child stealing.
Akiyama and Taura [28], [29] challenged the ﬁrst assumption by introducing an eﬃcient stack migration
scheme over RDMA that does not require special compiler support. However, the correctness of the second
assumption is still unclear; there have been no reports in the literature of a performance comparison between

34

4. Scalable Continuation Stealing on Distributed Memory
continuation stealing and child stealing on distributed memory. The primary goal of this chapter is to explore
the true potential of distributed continuation stealing by comparing its performance with that of child stealing.
To determine the full potential of distributed continuation stealing, we ﬁrst investigated eﬃcient task
synchronization (join) strategies over RDMA, which were not covered by previous work. In particular, theimplementation by Akiyama and Taura does not allow for migration of tasks suspended at a join. Because
they cannot be resumed by other processors even after they become ready for execution, available parallelism
is reduced. Thus, we devised an RDMA-based technique to migrate and resume tasks suspended at a join
immediately after they become ready, which is in fact a common practice in shared-memory implementations(e.g., Cilk). In addition, we devised an eﬃcient memory management technique for dynamically allocatedRDMA-accessible objects for task synchronization in continuation stealing. Our experiments showed thatthese join optimizations improved performance by up to 40% compared with the baseline.
Using a continuation-stealing runtime system implementing these two techniques, we conducted performance analysis of continuation stealing and child stealing. Although continuation stealing incurred longer
steal latency than child stealing, the average steal latency was less than 20% higher than that of child stealing.
Moreover, the experiments demonstrated that continuation stealing handled join primitives more eﬃciently,
resulting in better overall performance than child stealing when the task synchronization pattern was complicated.
Furthermore, we used the unbalanced tree search (UTS) benchmark [84] to compare our continuationstealing runtime implementation with existing task-parallel runtime systems (SAWS [56], Charm++ [85],[86], and X10/GLB [79], [87]). The results show that our runtime system was as scalable as or even morescalable than them. Notably, our continuation-stealing runtime system demonstrated 96.4% parallel eﬃciency
on 110,594 cores. This is a surprising result because our UTS implementation is based on a straightforward
fork-join parallelization of tree traversal, whereas other UTS implementations are based on a bag-of-tasks
paradigm, in which task dependency cannot be described and global termination detection is needed.
The task implementation of Itoyori is not only for fork-join primitives but also for more general futures. In
other words, tasks do not have to be joined with their parent; the handler of a spawned task, called a future, canbe passed to any other tasks. Thus, our runtime system can deal with a wider range of dependency patterns,
which is demonstrated by the longest common subsequence (LCS) benchmark used in our evaluation. Our
LCS benchmark makes heavy use of futures to represent the complicated dependency pattern (wavefront)
by recursively decomposing blocks. The evaluation results show that the capability of task migration (bothcontinuation stealing and task migration at joins) is indispensable to achieving good performance; lack of task
migration led to worse performance by an order of magnitude. Our continuation-stealing runtime system withboth task migration capabilities achieved the highest performance (close to the theoretical bounds) on nearly
10k cores.

4.2. BackgroundIn this section, we ﬁrst discuss child stealing and continuation stealing in Section 4.2.1 and demonstrate by
an example a shortcoming in child stealing when handling join primitives in Section 4.2.2. We then explain
the current status of support for thread migration on distributed memory in Section 4.2.3 and ﬁnally introduce
prior work on distributed continuation stealing, uni-address threads, in Section 4.2.4.

35

4. Scalable Continuation Stealing on Distributed Memory
thread T𝑝 = fork([=] {...
thread T𝑐 = fork([=] { B𝑐 ; });B𝑝;
5
T𝑐 .join();
6
...
7 });
1
2
3
4

(a) Pseudocode

W1 : parent ﬁrst B 𝑝

T𝑝

Spawn

W2 : steal

B𝑐

T𝑐

W2 : steal B 𝑝

T𝑝

Join

Spawn

W1 : child ﬁrst

T𝑐

Likely to block(b) Child stealing

B𝑐

Join
Unlikely to block(c) Continuation stealing

Figure 4.1.: Comparison of child stealing and continuation stealing. Parent task T𝑝 forks task T𝑐 , computes
B 𝑝 , and joins T𝑐 . Child task T𝑐 computes B𝑐 and exits. Worker W1 ﬁrst executes T𝑝 and moves
on to either B 𝑝 or T𝑐 when forking T𝑐 . W2 can steal either T𝑝 or T𝑐 left in the task queue.

4.2.1. Child Stealing and Continuation Stealing
As noted above, an important choice in work stealing is between child stealing or continuation stealing, which
are illustrated in Figure 4.1. Worker W1 ﬁrst runs task T𝑝 , which spawns child task T𝑐 . In child stealing, W1
continues executing T𝑝 after spawning T𝑐 and pushes T𝑐 to the local task queue. In continuation stealing, W1
ﬁrst executes T𝑐 upon spawning, leaving the continuation of T𝑝 subject to work stealing.
The beneﬁt of child stealing is its simple and portable implementation [88]. Each stealable task can be
simply represented as a function pointer (or a serialized closure) and its arguments because the task is stolen
before execution has begun. This simplicity is the reason many shared-memory tasking runtime systems use
child stealing [12], [18], [88], [89]. In contrast, the implementation of continuation stealing is more involved
because stealing the continuation of an already started task is not trivial. Many eﬀorts have been put into
compiler support for continuation stealing [8], [9], [90] and user-level threading libraries [15], [49] to saveand restore continuation states. Despite its implementation diﬃculties, continuation stealing nevertheless has
beneﬁts as explained in Section 2.2.2.

4.2.2. Handling Task Synchronization (Join)
How to join tasks is as important as how to steal tasks in task-parallel programs, but child stealing is problematicfor handling join. This is because child stealing is likely to be suspended at a join because the parent is preferably
executed rather than the child. As shown in Figure 4.1b, because the join for T𝑐 belongs to the parent, T𝑝 ,the parent-ﬁrst order is likely to reach the join point before the child is completed. Note that reaching anunresolved join itself is not a problem. Unless T𝑐 is stolen, W1 can immediately execute T𝑐 by popping itfrom the local task queue at the join point [88]. Then, once T𝑐 is completed, W1 returns to the join point andexecutes the continuation of T𝑝 . Along this path, the continuation of the join is executed immediately after it
becomes runnable.
A problem arises when T𝑐 is stolen by another worker (W2 in the ﬁgure). Supposing that B 𝑝 and B𝑐 have thesame amount of work, W1 is likely to complete B 𝑝 before W2 completes B𝑐 because of the steal delay. Thus,
W1 is likely to reach the join point before T𝑐 is completed. W1 thus initiates work stealing by suspending T𝑝

36

4. Scalable Continuation Stealing on Distributed Memory
at the join. We call such an unresolved join resulting from a steal event an outstanding join. An outstanding
join may not be resumable even after T𝑐 is completed by W2 if W1 is busy running another task and migrationof suspended tasks is impossible. In child stealing, it is typically impossible to migrate an already started task
to another worker, because of its simple task representation. Thus, child stealing can yield many outstanding
joins, most of which may not be immediately resumed, which can severely limit available parallelism.
Conversely, continuation stealing is unlikely to yield an outstanding join. Moreover, even if an outstanding
join appears, it can be immediately resumed because continuation stealing should already support dynamic
task migration.

4.2.3. Suspending and Migrating Threads
Even aside from work stealing, the generic suspension capability for tasks is important, as evidenced by theextensive research on the suspension capability of user-level threads on shared memory [2], [15], [45], [46],[48], [49]. The suspension capability is also useful for eﬃciently implementing a generic yield operation andother synchronization mechanisms (e.g., locks, barriers, and condition variables).
Many child-stealing runtime systems that operate on distributed memory, including Scioto [54], [55],
SAWS [56], and X10/GLB [79], [87], are based on the bag-of-tasks (BoT) paradigm and their tasks cannot
be suspended. Several distributed child-stealing runtime systems, including Satin [91], KAAPI [57], HotSLAW [58], and Grappa [59], support suspension at thread synchronization (e.g., at joins), but their thread
implementations are tied tasks, which means that a task cannot be migrated to another worker once it starts torun. Thus, they are also subject to the performance problem caused by outstanding joins. Compared with themany user-level thread implementations on shared memory, thread suspension on distributed memory has notbeen well investigated.

4.2.4. Uni-Address Threads
A naive approach to achieving thread migration across nodes, without special compiler support, is to copy athread stack to the same virtual address on the target node. The virtual address of the stack has to be preserved
because the stack may contain pointers to local stack variables. In the iso-address scheme [92], which is usedin Charm++ [85], [86] and Adaptive MPI [93], a globally unique virtual address is assigned to each stack inorder to prevent virtual address overlapping. However, this can consume a huge amount of virtual address
space, which is problematic in RDMA because memory for thread stacks would need to be pinned in advance.
To reduce address space consumption, the uni-address scheme was devised by Akiyama and Taura [28],[29]. The idea is to place thread stacks at the same virtual address only while threads are running. When athread is suspended, its stack is evacuated to an arbitrary virtual address to make room for another thread torun. When the suspended thread is resumed, its stack is brought back to the virtual address to which it wasﬁrst allocated. Therefore, the virtual addresses of thread stacks are no longer globally unique, reducing overall
virtual address consumption. Note that this does not mean that threads at the same virtual address cannot be
executed at the same time as each worker has its own address space. We call a memory region for running
threads a uni-address region and one for suspended threads an evacuation region. Both regions are pinned
to physical memory accessible by RDMA. Workers need to have the same virtual address for the uni-address
region, but not for the evacuation region.
One key point of the uni-address scheme is that it is optimized for continuation stealing. To avoid frequentcopying of thread stacks between the uni-address region and the evacuation region, a new stack is placed on37

4. Scalable Continuation Stealing on Distributed Memory

Virtual address

Uni-address region Uni-address region

Virtual address

Uni-address region Uni-address region

T𝐶 2. Die
1. Fork T𝐵
T𝐴 3. Steal T𝐴
W1

T𝐸

Possible?

T𝐷

5. Resume

Evacuation region Evacuation region

4. Suspend T𝐷

T𝐸

W1

W2

W2

Figure 4.2.: Stack management in uni-address threads.
top of the current thread stack so that the virtual addresses of thread stacks with a parent-child relationship do
not overlap. Figure 4.2 illustrates the ﬂow of continuation stealing with uni-address threads.
1. Fork When T𝐵 creates a new thread T𝐶 , T𝐶 ’s stack is placed immediately above T𝐵 ’s stack. The virtual

address and size of T𝐵 ’s stack are recorded in the task queue to be potentially stolen.
2. Die When T𝐶 is completed, the execution of T𝐵 is resumed in the same way as an ordinary subroutine

returns, unless T𝐵 has already been stolen by another worker.
3. Steal When W2 steals T𝐴 from W1 , the call stack of T𝐴 at W1 is copied to the same virtual address at W2 ,preserving pointers to local stack variables.
4. Suspend When T𝐷 is blocked due to an unresolved join, for example, T𝐷 ’s call stack is temporarily moved

to any virtual address in the evacuation region.
5. Resume When T𝐸 is resumed, T𝐸 ’s stack is moved from the evacuation region to the virtual address towhich it was previously assigned.
The original uni-address thread implementation has shortcomings at joins; e.g., thread migration at joins(indicated by the dashed arrow in Figure 4.2) is not allowed.

4.3. Joining Threads over RDMA
As brieﬂy mentioned in Section 4.2.4, eﬃcient join implementations on distributed memory have not been
well investigated. This section ﬁrst examines diﬀerent joining strategies in Section 4.3.1 and then proposes an
eﬃcient memory management technique in Section 4.3.2.

4.3.1. Joining Strategy
In general, joining strategies for work stealing can be classiﬁed as either stalling or greedy [94]. In the stalling
strategy, threads are not migrated to other workers across a join, whereas in the greedy strategy, threads canbe migrated to other workers so that they are resumed immediately after becoming ready. On shared memory,for example, Intel TBB [12] uses the stalling strategy and Cilk uses the greedy strategy.
We introduce several notations to explain our algorithms. The location of a variable speciﬁes where thevariable is; it typically consists of the worker ID (rank) of the owner, the virtual address, and the size. Type
Loc(𝑇) speciﬁes the location of a variable of type 𝑇. The “get 𝑣 ← 𝐿” statement copies the remote variable at38

4. Scalable Continuation Stealing on Distributed Memory

1 Struct ThreadEntry

retval :: return value of the joined thread function
ﬂag :: completion status of the joined thread (default: 0)
4 Function Die(𝐸 :: Loc(ThreadEntry), retval)
5
put 𝐸.retval ← retval
6
put 𝐸.ﬂag ← 1
7
nextThread ← PopFromThreadQueue()8
if nextThread is found then9
resume nextThread.context
10
else
11
resume scheduler.context
12 Function Join(𝐸 :: Loc(ThreadEntry))
get 𝑓 ← 𝐸 .ﬂag
1314
while 𝑓 = 0 do // The joined thread has not been completed.
15
suspend context do
16
PushToWaitQueue(𝑐𝑜𝑛𝑡𝑒𝑥𝑡)
17
resume scheduler.context
18
get 𝑓 ← 𝐸.ﬂag
19
get retval ← 𝐸.retval
20
FreeRemote(𝐸)21return retval
2
3

Figure 4.3.: Implementation of stalling join over RDMA.
location 𝐿 to local variable 𝑣, and the “put 𝐿 ← 𝑣” statement copies local variable 𝑣 to the remote variable at
𝐿. The “fetch and add (𝐿, 𝑣)” statement atomically adds 𝑣 to the remote variable at 𝐿 and returns the originalvalue. The “Suspend 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 do” block ﬁrst saves the current execution state to 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 and then executes
the statements in the block on a separate context (stack). The saved context can be resumed by executing the“resume 𝑐𝑜𝑛𝑡𝑒𝑥𝑡” statement, which discards the current execution context.
Stalling Join

Figure 4.3 shows the stalling join implementation in the original uni-address threads [28]. In the following,
without loss of generality, we consider only two threads: a joining thread and a joined thread. The joining
thread waits for completion of the joined thread by the Join function, and the joined thread calls the Die
function when completed. In continuation stealing, the joining thread does not know where the joined thread
is, and vice versa, as threads can be dynamically migrated. Thus, we allocate a thread entry to the memory
where the joined thread was originally spawned, so that the joined thread and joining thread can communicate
via this RDMA-accessible thread entry. A thread entry’s location is used as a thread handler, and both the Die
and Join function receive it as an argument. The type ThreadEntry has two ﬁelds: retval, the return valueof the thread, and ﬂag, which indicates the exit status of the joined thread.
When the joined thread is completed, it ﬁrst puts the return value (line 5) and then sets the ﬂag in the thread
entry (line 6). The joining thread waits for the ﬂag to be set in a loop (line 14–18), after which it gets thereturn value of the joined thread (line 19). Each time the ﬂag check fails, it switches to the scheduler context
to initiate work stealing; at the same time, its current context is saved and pushed into the wait queue. A wait
queue is a per-worker FIFO queue in which suspended threads are stored. Each time a steal attempt fails, the39

4. Scalable Continuation Stealing on Distributed Memory

22 Struct ThreadEntry

retval :: return value of the joined thread function
ﬂag :: atomic ﬂag for greedy join (default: 0)25
ctxloc :: location of the saved context of the joining thread
26 Function Die(𝐸 :: Loc(ThreadEntry), retval)
27
put 𝐸.retval ← retval
28
nextThread ← PopFromThreadQueue()29
if nextThread is found then // The parent has not been stolen.
30
put 𝐸 .ﬂag ← 1
31resume nextThread.context
32
else // The parent has already been stolen.
33𝑓 ← fetch and add (𝐸 .ﬂag, 1)
34
if 𝑓 = 0 then // The joined thread won the race.
35
resume scheduler.context
36
else // The joined thread lost the race.
37
get 𝐶 ← 𝐸 .ctxloc
38
get 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 ← 𝐶
39
FreeRemote(𝐶)40
resume context
41 Function Join(𝐸 :: Loc(ThreadEntry))42
get 𝑓 ← 𝐸 . 𝑓 𝑙𝑎𝑔
43
if 𝑓 = 0 then // The joined thread seems to be running.
44
suspend context do
45
put 𝐸.ctxloc ← LocationOf(context)46
𝑓 ← fetch and add (𝐸 .ﬂag, 1)
47
if 𝑓 = 0 then // The joining thread won the race.
48
resume scheduler.context
49
else // The joining thread lost the race.
50
resume context
51
get retval ← 𝐸.retval
52
FreeRemote(𝐸)53return retval
23
24

Figure 4.4.: Implementation of greedy join over RDMA.
scheduler pops a thread from the wait queue and resumes its execution in a round-robin fashion. A suspended
thread cannot be executed until it is popped from the wait queue, even if it becomes runnable, which makes
this scheduler nongreedy.
Greedy Join

In the greedy join implementation, the continuation of a join is executed by the worker who runs the joined
thread or the joining thread, whichever reaches the synchronization point later. Therefore, the continuation ofa join is executed immediately after it becomes runnable. While Cilk uses a mutex for implementing greedy
join, the use of atomic operations on shared memory has been investigated [95], [96]. Borrowing this idea, weconsider an eﬃcient greedy join implementation using RDMA atomic operations.
Figure 4.4 shows our greedy join implementation. The basic idea is that the two threads race on an atomic

40

4. Scalable Continuation Stealing on Distributed Memory
variable, and the worker who runs the loser (who executes the atomic operation later) runs the continuationof the join. However, calling an RDMA atomic operation at every join and die can be too costly. To avoid
frequent atomic operations, we devised a technique based on the work-ﬁrst principle [9]. The idea is for thejoined thread to try to pop the parent thread from the queue (line 28) before racing with the joining thread.
Because we assume continuation stealing and the oldest-ﬁrst stealing strategy of the queue, the popped thread
is guaranteed to always be the parent of the joined (completed) thread [26]. If the parent thread is successfully
popped, the corresponding join is guaranteed to occur only after the joined thread has died. Therefore, we cansimply set the ﬂag without atomic operations (line 29–31), which is in turn read by the same worker at line 43.
Along this fast path, the Join function can be completed without suspending its execution.
If the parent thread has already been stolen when the joined thread is completed, the execution moves tothe slow path (line 32–40). Because the return value has been written at line 27, an atomic fetch-and-add isexecuted to race with the joining thread (line 33). If the joined thread wins the race, it simply returns to thescheduler to initiate work stealing (line 34–35). Otherwise, it resumes the continuation of the join (line 36–40)
because the joining thread has already been suspended at the join. The joining thread performs the atomic
fetch-and-add (line 46) after saving its current context and putting its location in the thread entry (line 45) sothat the joined thread can obtain the context immediately after the race (line 37–38). On rare occasions, thejoining thread loses the race even after it sees that the ﬂag is unset (line 43) and suspends its execution, in
which case it resumes the continuation of the join by itself (line 50).
Note that this algorithm is not limited to fork-join but is also applicable to futures [97], [98]; the joining
thread does not need to be the parent of the joined thread. One limitation in this implementation is that a future
can have only one consumer, but this limitation will be slightly relaxed in Section 4.5.4.

4.3.2. Freeing Remote Objects
In continuation stealing, thread entries and suspended threads can be freed remotely by any worker; we call such
RDMA-accessible objects remote objects. The FreeRemote function in Figure 4.3 and Figure 4.4 receivesa remote object’s location and frees it (possibly) remotely. In the original implementation by Akiyama andTaura, each worker has its own lock-protected incoming queue to receive locations of remotely freed objects.
When freeing an object remotely, it acquires the lock of the target queue, increments the counter, inserts theobject location into the buﬀer, and releases the lock; this operation involves four round trips. When a worker
collects remotely freed objects in its local queue, it acquires the lock, frees all received objects, sets the counterto zero, and releases the lock.
The key to improved performance is to move the overheads of remote workers to the local worker because
the cost of local operations is much lower than that of remote operations. Thus, we introduce an optimization
called local collection. In local collection, all remote objects are managed in a doubly linked list by the local
worker. When a remote object is newly allocated, it is added to the list. When a remote object is freed locally,
it is immediately removed from the list and freed. When it is freed remotely, its free bit is set by a remote
put operation in a nonblocking manner. When the total size of the allocated remote objects exceeds a limit,
the worker iterates over the list and frees remote objects with a free bit set. This strategy eliminates locks andinvolves only one communication for freeing a remote object (which can even be nonblocking), which greatly
reduces the cost of remote workers.

41

4. Scalable Continuation Stealing on Distributed Memory

Table 4.1.: Experimental environment for distributed multithreading.

Processor# of Physical Cores
CPU Frequency
MemoryInterconnect
CompilerMPIOS

ITO-A [99]Wisteria-O [100]Intel Xeon Gold 6154 (Skylake-SP)
36 (18 cores/socket × 2 sockets)
3.0 GHz (Turbo 3.7 GHz)
192 GB
InﬁniBand EDR 4x (100 Gbps)
GCC 11.2.0 (-O3 -march=native)
Open MPI 5.0.x
Red Hat Enterprise Linux 7.3

Fujitsu A64FX (ARMv8.2-A + SVE)
48
2.2 GHz
32 GB
Tofu Interconnect D
Fujitsu Compiler 4.5.0 (-O3 -Nclang)
Fujitsu MPI 4.0.1
Red Hat Enterprise Linux 8.3

4.4. Evaluation Methodology
Our primary interest here is performance comparison between diﬀerent joining strategies (stalling vs. greedyjoin) and stealing strategies (child vs. continuation stealing). To analyze their performance, we implemented
child stealing in our runtime library by mimicking the prevalent distributed child-stealing implementations(Section 4.4.2), in addition to continuation stealing with both stalling and greedy join. We use synthetic
benchmarks (Section 4.4.3) to explore the detailed performance characteristics and more realistic applications(the UTS and LCS benchmarks) for evaluation.

4.4.1. Experimental Settings
Our implementation is based on a C++ library MassiveThreads/DM (an implementation of uni-address threads)
originally developed by Akiyama and Taura [28], [29]. To achieve better portability, we modiﬁed it to use
MPI-3 RMA [40] as a backend communication library, as high-quality implementations for MPI-3 RMA are
available today. In addition, we ﬁxed a few trivial performance issues in MassiveThreads/DM and used it as
the baseline for evaluation.
The experimental environment is summarized in Table 7.1. ITO-A is ITO supercomputer (subsystem A)
at Kyushu University [99], which is comprised of Intel Skylake processors and InﬁniBand Interconnect,
and Wisteria-O is Wisteria/BDEC-01 supercomputer (Odyssey subsystem) at the University of Tokyo [100],
comprised of Fujitsu A64FX processors and Tofu interconnect D developed for Fugaku supercomputer [101].
For ITO-A, we built a nightly version (5.0.x) of Open MPI (Git commit hash: 85fec3a74fee) with UCX [102]1.11.0 and the osc ucx acc single intrinsic option; it has better one-sided communication performance
than other MPI implementations [41]. When allocating nodes in Wisteria-O, we speciﬁed a 3D mesh topology
as close to a cube as possible, because it showed the best performance for random work stealing. To obtainstable performance results, we executed warm-up runs until the performance results stabilized. Each pointin the following plots represents an average of 100 runs with an error bar representing the 95% conﬁdence
interval unless explicitly noted.

4.4.2. Implementations of Child Stealing
Assuming that thread implementations in child stealing are tied tasks as discussed in Chapter 2, we implemented
only stalling join for child stealing. We further classify stalling join implementations into two types dependingon the thread suspension capability, run-to-completion (RtC) threads and fully ﬂedged (Full) threads, following

42

4. Scalable Continuation Stealing on Distributed Memory
PFor(int n) {for (int k = 0; k < K; k++) {parallel_for (int i = 0; i < n; i++)compute(M); // run for duration of M
5
}
6 }
1
2
3
4

1
2
3
4
5
6
7
8
9
10

RecPFor(int n) {if (n == 1) {compute(M);} else {PFor(n);thread th = fork([=]{ RecPFor(n/2); });RecPFor(n/2);
th.join();}}Figure 4.5.: Pseudocode of PFor and RecPFor benchmarks.
the taxonomy of Iwasaki et al. [48], [49].
RtC threads are simply realized by ordinary function calls. When a RtC thread encounters an unresolved
join, it calls the scheduler function directly on top of its stack to ﬁnd another runnable task. If a runnable task
is found, the task is executed on top of the stack as in ordinary function calls. Meanwhile, the unresolved join
is “buried” and cannot be resumed until all the tasks running on top of it are completed. This “buried join”
problem [89] can substantially reduce available parallelism.
Full threads avoid the buried join problem because each thread has its own stack. When a task begins,
it is assigned a new separate stack (32 KB in our experiments). When encountering an unresolved join, it
context-switches to another thread by suspending itself (using assembly instructions). Suspended threads aremanaged in the wait queue (Section 4.3.1).

4.4.3. Synthetic Benchmarks
Figure 4.5 shows the pseudocode of the synthetic benchmarks we used for performance analysis: PForand RecPFor. In the PFor benchmark, a simple parallel loop is repeated multiple times. Each parallel loop
is implemented as a recursive binary fork-join pattern (as in cilk_for in Cilk Plus [103]). In the RecPFor
benchmark, parallel tasks are recursively created as a binary tree, and, at each recursion, parallel loops arerepeatedly executed. This computation pattern appears in many algorithms (e.g., quicksort and decision tree
construction [104]) in which an input array is ﬁrst manipulated by parallel loops and then divided into two
sub-arrays, and the same procedure is applied to each sub-array recursively. The RecPFor benchmark mimics
this behavior.
These benchmarks are parameterized by the number of consecutive parallel loops in the PFor() function(𝐾), the execution time of each leaf task (compute() function) (𝑀), and the problem size given to the root
task (𝑁). The compute() function executes a ﬁxed number of SIMD fused multiply-add (FMA) operations.
The number of FMA operations was conﬁgured to run for a speciﬁed duration (𝑀). The total work of the PFor
benchmark is 𝑇1 = 𝐾 𝑀 𝑁, and that of the RecPFor benchmark is 𝑇1 = 𝐾 𝑀 𝑁 log2 𝑁 + 𝑀 𝑁. For evaluation, we
ﬁxed 𝐾 = 5 and 𝑀 = 10𝜇s for both benchmarks and varied 𝑁 to change the problem size.

43

4. Scalable Continuation Stealing on Distributed Memory

Cont. Steal (local collection + greedy join)Cont. Steal (local collection)
Child Steal (RtC)

100

100

90

90

80

80

Parallel Efficiency (%)

Parallel Efficiency (%)

Child Steal (Full)

70
60
50
40
30
20
10

70
60
50
40
30
20
10

0

0
10 ms

100 ms1s

100 ms

Ideal Execution Time

1s

10 s

Ideal Execution Time(a) PFor on ITO-A(b) RecPFor on ITO-A

100

100

90

90

80

80

Parallel Efficiency (%)

Parallel Efficiency (%)

Cont. Steal (baseline)

70
60
50
40
30
20
10

70
60
50
40
30
20
10

0

0
10 ms

100 ms1s

100 ms

Ideal Execution Time

1s

Ideal Execution Time(c) PFor on Wisteria-O(d) RecPFor on Wisteria-O

Figure 4.6.: Performance comparison between diﬀerent joining strategies and steal strategies with synthetic
benchmarks (PFor and RecPFor). The number of cores on ITO-A was 576 cores (16 nodes) andthat on Wisteria-O was 1,728 cores (36 nodes). X-axis represents ideal execution time calculated
by 𝑇1 /𝑃 (see Section 4.4.3); y-axis represents parallel eﬃciency (ratio of ideal execution time tomeasured execution time).

44

4. Scalable Continuation Stealing on Distributed Memory

4.5. Performance Analysis and Evaluation
4.5.1. Performance Analysis of Joining Strategies
To evaluate the join techniques introduced in Section 4.3, we implemented a version with only local collection
enabled and a version with both local collection and greedy join enabled. The baseline is the original
implementation of MassiveThreads/DM with a few trivial performance issues ﬁxed.
The parallel eﬃciencies of the synthetic benchmarks with various problem sizes (parameter 𝑁) are plotted in
Figure 4.6. The ideal execution time is simply 𝑇1 /𝑃, where 𝑇1 is the theoretical work discussed in Section 4.4.3,
and parallel eﬃciency is the ratio of the ideal execution time to the actual (measured) execution time. In PForon ITO-A, local collection achieved up to 40% improvement over the baseline. Greedy join did not contributeto the performance improvement, whereas in RecPFor it did. This is because RecPFor has a more complicatedsynchronization pattern than PFor; the PFor benchmark has little work after a join at each recursion, while the
RecPFor benchmark has substantial work after each join as parallel loops are repeated at each recursion. As
a result, in RecPFor on ITO-A, local collection improved performance by 27% over the baseline, and greedy
join improved performance by an additional 8%. In total, our join techniques achieved up to 37% performance
improvement over the baseline in RecPFor on ITO-A. We observed similar results for Wisteria-O.

4.5.2. Child Stealing vs. Continuation Stealing
Figure 4.6 also shows the performance of child stealing with Full and RtC threads. In PFor, the performance
diﬀerence between child and continuation stealing is small, whereas in RecPFor, continuation stealing consistently outperformed child stealing. Notably, continuation stealing was at most 1.3× faster than child stealing(Full) and at most 4.8× faster than child stealing (RtC) in RecPFor on Wisteria-O. Note that Full threads incurlarger overheads on Wisteria-O because of their relatively large context switching costs and slower processor
speed.
Table 4.2 shows statistics for outstanding joins and steal events, which were proﬁled using the largest
problem sizes in Figure 4.6. The numbers of outstanding joins and successful steals show that, in childstealing, a majority of successful steals resulted in outstanding joins. The numbers for child stealing (RtC)
are exceptionally large because of the buried join problem explained in Section 4.4.2. Continuation stealing
incurred by an order of magnitude fewer outstanding joins than successful steals. This supports our assertionin Section 4.2.2 that continuation stealing causes fewer outstanding joins.
Here, we introduce a metric, outstanding join time, which is the duration since an outstanding join’s
continuation became resumable (i.e., both the joining and joined thread reach the synchronization point) anduntil it is actually resumed. This metric well characterizes the join implementations as all three implementations
based on stalling join have long average outstanding join times, whereas continuation stealing (greedy) has
the shortest outstanding join time. Note that a longer outstanding join time does not necessarily mean that thescheduling is bad; for example, as long as all of the workers are busy executing tasks, a long outstanding jointime is not a problem. However, it becomes problematic when workers are idle, because idle workers should
execute ready tasks (outstanding joins) but do not execute them, which is attributable to the scheduler [105].
To conﬁrm that outstanding joins are related to the low eﬃciency of child stealing, we plotted time series
data for RecPFor in Figure 4.7. The ﬁlled (green) area shows how many workers were busy executing tasks,
and the line (purple) plot represents the number of outstanding joins that were runnable at that time. With
continuation stealing, almost all workers were busy all the time, and there were almost no outstanding joins.
In contrast, with child stealing (Full), the number of busy workers has many “valleys” in the latter half of the45

Wisteria-O

ITO-A

System
8672
6832
45920
61005
56876
72546
320841740797378
25425
19093
1561161758452291542790358602558205358008

2.98 s2.98 s3.07 s3.01 s8.30 s8.33 s9.31 s
48.64 s2.30 s2.30 s2.57 s2.37 s
5.94 s
5.97 s7.69 s
27.71 sCont. Steal (greedy)Cont. Steal (stalling)Child Steal (Full)
Child Steal (RtC)

Cont. Steal (greedy)Cont. Steal (stalling)Child Steal (Full)
Child Steal (RtC)

Cont. Steal (greedy)Cont. Steal (stalling)Child Steal (Full)
Child Steal (RtC)

Cont. Steal (greedy)Cont. Steal (stalling)Child Steal (Full)
Child Steal (RtC)

RecPForPFor

RecPForPFor# of
Outstanding
Joins

Steal Strategy

Bench

ExecutionTime# of Steals(Successful)
72831
74091
76583
75777
474991807097603885845092301
20685721060122188721842017900183086034
15704498
223562173

Avg.
Outstanding
Join Time14.1 𝜇s
13965.4 𝜇s
1016.3 𝜇s
17096.3 𝜇s
15.0 𝜇s
8128.6 𝜇s
8237.9 𝜇s
297.6 𝜇s
6.6 𝜇s7006.5 𝜇s
165.2 𝜇s
8887.9 𝜇s

46
6.5 𝜇s
4852.7 𝜇s6309.9 𝜇s
88.9 𝜇s20.4 𝜇s20.6 𝜇s19.9 𝜇s19.2 𝜇s20.7 𝜇s20.9 𝜇s19.8 𝜇s
18.4 𝜇s

31.6 𝜇s
30.4 𝜇s29.3 𝜇s59.5 𝜇s28.8 𝜇s
28.9 𝜇s27.7 𝜇s27.1 𝜇s

Avg. Steal
Latency(Successful)

31861164349765625091108164463070

1602730
1583784134105615718702853760
2561299
16656796
1471927098

1108412108957849362102661418# of Steals(Failed)1139 bytes
1156 bytes
55 bytes
56 bytes

1289 bytes
1286 bytes
56 bytes
56 bytes

1845 bytes
1790 bytes
55 bytes
55 bytes

1789 bytes
1687 bytes
56 bytes
56 bytes

Avg. Stolen
Task Size

3.37 𝜇s
3.42 𝜇s2.90 𝜇s
2.98 𝜇s

3.47 𝜇s
3.50 𝜇s
2.91 𝜇s
2.78 𝜇s

5.72 𝜇s
5.68 𝜇s
4.18 𝜇s
8.53 𝜇s

5.85 𝜇s
6.02 𝜇s3.76 𝜇s
3.83 𝜇s

Avg. Task
Copy Time

Table 4.2.: Statistics of join and steal events in child stealing and continuation stealing. The number of cores used on ITO-A was 576 cores (16 nodes) andthat on Wisteria-O was 1,728 cores (36 nodes). The problem sizes correspond to the largest sizes in Figure 4.6.

4. Scalable Continuation Stealing on Distributed Memory

500

50k

400

40k

300

30k

600
400
200
0

200
100

20k
10k

0

0
0

1

2

3

4

5

6

Time (s)78# of Outstanding Joins# of Busy Workers

4. Scalable Continuation Stealing on Distributed Memory9# of outstanding joins

500

50k

400

40k

300

30k

20

200

10

20k

100

0

10k

0

0
0

1

2

3

4

5

6

7

8

# of Outstanding Joins# of Busy Workers(a) Child Stealing (Full)

9Time (s)(b) Continuation Stealing (greedy join)

Figure 4.7.: Time series of scheduler activities in RecPFor benchmark (𝑁 = 222 ) on ITO-A with 576 cores(16 nodes). Filled (green) area represents number of busy workers (left y-axis); line (purple) plot
represents number of ready-to-execute outstanding joins (right y-axis). Numbers of outstanding
joins between 1 s and 5 s are zoomed in and plotted as an overlay on the main chart.
execution. Simultaneously, there were a large number of outstanding joins, which means that runnable tasks
were not being executed despite some workers being idle because of a nongreedy schedule.
Finally, we analyze steal latency. Table 4.2 shows that continuation stealing had two orders of magnitudelarger “stolen task sizes” than child stealing because of stack migration. Therefore, continuation stealing
incurs a longer “task copy time,” but the times are of the same order of magnitude. Note that child stealing(RtC) in RecPFor on ITO-A incurred much longer latency because of the greater number of steal attempts,
whereas latency on Wisteria-O was less aﬀected. Overall, we found that the increased average steal latency(successful) with continuation stealing was less than 20% compared with that for child stealing for all cases in
Figure 4.6. Summarizing, despite a small increase in steal latency, continuation stealing can outperform childstealing by eﬃciently resolving joins.

4.5.3. Unbalanced Tree Search (UTS)The UTS benchmark [84] has been used for measuring the load balancing capability of parallel runtime systems
in many studies [28], [29], [55], [56], [58], [79], [83], [87], [106]–[108]. Most existing implementations on
distributed memory are based on a BoT, in which task dependency cannot be described. We found that thescalability of our runtime system was comparable to or even better than that of existing state-of-the-art UTS
implementations.

47

4. Scalable Continuation Stealing on Distributed Memory

Ours (T1WL)
Ours (T1XXL)
Ours (T1L)

SAWS (T1WL)SAWS (T1XXL)
SAWS (T1L)

100G

Throughput (nodes/s)

Throughput (nodes/s)

100G

10G

1G

10G

1G

100M

100M
100

1K

10K

100

# of processes(a) Itoyori (ours).

10G

Throughput (nodes/s)

Throughput (nodes/s)

X10 (T1WL)
X10 (T1XXL)
X10 (T1L)

100G

1G
100M
Charm++ (T1WL)
Charm++ (T1XXL)
Charm++ (T1L)

1M
100

1K

10K

(b) SAWS.

100G

10M

1K

# of processes

10G

1G

100M
10K

100

# of processes

1K# of processes(c) Charm++.

(d) X10/GLB.

Figure 4.8.: Throughput of UTS benchmark on ITO-A (up to 9,212 cores).

48

10K

4. Scalable Continuation Stealing on Distributed Memory
The task of UTS is to count the total number of nodes in an unbalanced tree, which is generated on theﬂy using a random hash (SHA-1). The identical tree is generated deterministically if the same seed andparameters are given. A natural way to count the number of nodes would be depth-ﬁrst tree traversal, andrecursive fork-join constructs can straightforwardly parallelize the tree traversal.
We compared the performance of Itoyori’s multithreading layer (ityr::ito namespace) with three existing
distributed work-stealing implementations: SAWS [56], Charm++ [85], [86], and X10/GLB [79], [87]. SAWS
is a state-of-the-art RDMA-based work-stealing library (see Chapter 8). We used the author-provided implementation of SAWS and UTS [109] with the OpenSHMEM implementation build together with Open MPI in
our environment (Table 7.1). We compiled Charm++ v7.0.0 with the MPI backend and the ParSSSE [108] implementation of UTS included in the oﬃcial repository, with work stealing enabled (-module workstealing
option). We built X10 [110] with the MPI backend and the GLB-UTS benchmark [111] using the compiler
with the C++ backend (x10c++). We modiﬁed all of their UTS implementations to run multiple times to
execute the same number of warm-up runs. The implementations were based on a BoT1 , which relies on global
termination detection and collective communication to gather the node counts at the end of execution.
For evaluation, we used three sizes of the geometric tree (T1L < T1XXL < T1WL). The primary performance
metric was throughput, which was calculated in terms of the number of nodes visited per second (nodes/s).
The execution time of the serial depth-ﬁrst search with T1L was 19.4 s (5.27 Mnodes/s) on ITO-A and 65.9 s(1.55 Mnodes/s) on Wisteria-O. Note that Itoyori adds serial overheads of only 18% on ITO-A and 27% on
Wisteria-O to the serial depth-ﬁrst execution.
Figure 4.8 shows the scalability of the four UTS implementations on ITO-A. The dotted straight line
represents the ideal throughput calculated using the throughput of a serial depth-ﬁrst search. Although thethroughput of X10/GLB exceeded the ideal because the UTS implementation of X10/GLB is highly optimized
from the original, we do not delve deeper into its serial performance because scalability is our primaryinterest. We can see that Charm++ did not scale to large core counts and that X10/GLB failed to exhibit
strong scaling for small trees. We consider that these performance problems were due to the message-based(two-sided) work-stealing implementation in Charm++ and X10/GLB. In contrast, the RDMA-based (onesided) implementations, Itoyori and SAWS, exhibited good scalability even for relatively small trees. Note that
the UTS implementation of SAWS was based on a BoT, which is a more restrictive form of parallelism than
the nested fork-join in Itoyori. These results indicate that continuation stealing (thread migration) and task
dependencies (join) are not performance-limiting factors in implementing distributed task-parallel runtime
systems.
Figure 4.9 shows the scalability of Itoyori on Wisteria-O, for which the trend is similar to that on ITO-A.
We did not run other runtime systems on Wisteria-O because of porting issues. Notably, for T1WL on
110,592 cores, it achieved 96.4% parallel eﬃciency calculated with a single-core execution time.

4.5.4. Longest Common Subsequence (LCS)
To examine scheduler performance when futures were intensively used, we developed the LCS benchmark. The
LCS problem for two sequences 𝐴 = ⟨𝑎 1 , ..., 𝑎 𝑛 ⟩ and 𝐵 = ⟨𝑏 1 , ..., 𝑏 𝑛 ⟩ is deﬁned by the following recurrence:


0




𝑋 (𝑖, 𝑗) = 𝑋 (𝑖 − 1, 𝑗 − 1) max{𝑋 (𝑖, 𝑗 − 1), 𝑋 (𝑖 − 1, 𝑗)}(𝑖 = 0 ∨ 𝑗 = 0)(𝑖, 𝑗 > 0 ∧ 𝑎 𝑖 = 𝑏 𝑗 )(𝑖, 𝑗 > 0 ∧ 𝑎 𝑖 ≠ 𝑏 𝑗 )1 Although the UTS implementation of Charm++ is based on a BoT, Charm++ itself has more synchronization capability than a BoT.

49

Throughput (nodes/s)

4. Scalable Continuation Stealing on Distributed Memory
Ours (T1WL)
Ours (T1XXL)
Ours (T1L)

100G

At leaf

𝑟

𝑙

10G

𝑏
𝐿

1G

100M
100

1K

10K

𝑇

𝑡

100K

# of processes
Figure 4.9.: Throughput of UTS with Itoyori on
Wisteria-O.

𝑇00

𝑇01

𝑇10

𝑇11

𝐿 00

𝐿 01

𝑋00

𝑋01

𝐿 10

𝐿 11

𝑋10

𝑋11

Figure 4.10.: Task dependency of LCS.

54 Function LCS(𝑖, 𝑗, 𝑇, 𝐿, 𝑛)
55
56
57
58
59
60
61
62
63
64
65
66

if 𝑛 ≤ 𝐶 then(𝑡, ) ← 𝑇 .join(), ( , 𝑙) ← 𝐿.join()(𝑏, 𝑟) ← LCS SEQ(𝑖, 𝑗, 𝑡, 𝑙, 𝑛)
return (𝑏, 𝑟)else( , 𝑇10 , 𝑇11 ) ← 𝑇 .join(), (𝐿 01 , , 𝐿 11 ) ← 𝐿.join()
𝑋00 ← fork LCS(𝑖,𝑗, 𝑇10 , 𝐿 01 , 𝑛/2)
𝑋01 ← fork LCS(𝑖, 𝑗 + 𝑛/2, 𝑇11 , 𝑋00 , 𝑛/2)
𝑋10 ← fork LCS(𝑖 + 𝑛/2, 𝑗, 𝑋00 , 𝐿 11 , 𝑛/2)
𝑋11 ← fork LCS(𝑖 + 𝑛/2, 𝑗 + 𝑛/2, 𝑋01 , 𝑋10 , 𝑛/2)
𝑋00 .join()return (𝑋01 , 𝑋10 , 𝑋11 )Figure 4.11.: Algorithm for LCS based on recursive decomposition and futures.

𝑋 (𝑛, 𝑛) is the length of the LCS for 𝐴 and 𝐵. Although a naive algorithm would require 𝑂 (𝑛2 ) space forcomputing the LCS, linear space algorithms with 𝑂 (𝑛) space are known [112], [113]. In this evaluation,
following the problem setting in [114], [115], we solved only for the length of an LCS. LCS computation canbe naturally parallelized by divide-and-conquer and has good cache locality, but a naive divide-and-conquer
approach lengthens the critical path from 𝑂 (𝑛) to 𝑂 (𝑛log2 3 ), which severely degrades parallelism [112], [113].
This is because strict fork-join programs add artiﬁcial dependencies to the original wavefront dependency
pattern. Although our approach is also based on recursive decomposition of 2D space similar to the naive
divide-and-conquer approach, the use of futures resolves the critical path length problem.
Figure 4.10 illustrates the recursive 2D decomposition and task dependencies between the blocks; thenotations correspond to those used in the algorithm in Figure 4.11, which is based on the sequential algorithm
by Chowdhury and Ramachandran [113]. Futures are dynamically created at each recursion and form a treelike structure in which each node is a future. Each intermediate node has three child futures (𝑋01 , 𝑋10 , 𝑋11 ),and each leaf has the values at the boundaries (𝑏, 𝑟) to be passed to the successive computations. The LCS

50

4. Scalable Continuation Stealing on Distributed Memory

Table 4.3.: Execution times of LCS on ITO-A with 576 cores (16 nodes).
Cont. Steal (greedy)Cont. Steal (stalling)Child Stealing (Full)

218
222

0.569 s45.9 s3.44 s433 s93.1 s2.11 ×104 s

1,000

Execution time (s)Execution time (s)

Size

100
10
1

1,000

100

10

1
10

100

1K

10K

10

# of processes

100

1K

10K

100K

# of processes(a) ITO-A(b) Wisteria-O

Figure 4.12.: Execution times of LCS with continuation stealing (greedy join). The ﬁlled area enclosed bylines represent max(𝑇1 /𝑃, 𝑇∞ ) ≤ 𝑇𝑃 ≤ 𝑇1 /𝑃 + 𝑇∞ (lower bound and upper bound of the ideal,
greedy-scheduling theorem) for each problem size.
function receives two futures, 𝑇 and 𝐿, which are geometrically at the top and left of the current block. At
an intermediate level, we ﬁrst join them (line 60) and the returned child futures (𝑇10 , 𝑇11 , 𝐿 01 , 𝐿 11 ) are thenpassed to the child computations (line 61–64), following the dependency pattern illustrated in Figure 4.10. 𝑋00
is joined at line 65 to avoid creating an excessive number of futures. When problem size 𝑛 is suﬃciently small(𝑛 < 𝐶), the algorithm sequentially computes the values at the output boundaries (𝑏 and 𝑟) using the values at
the input boundaries (𝑡 and 𝑙), which are acquired by joining the futures (𝑇 and 𝐿).
We experimentally implemented futures that allow for multiple consumers in our runtime system. A
limitation is that we must set a ﬁxed number of consumers when spawning a future, but it suﬃces for LCS
because the number of consumers is known in advance. When a future is completed and it ﬁnds multiple
futures waiting for its completion, it immediately resumes one of them and pushes the others into the local
task queue. For evaluation, we set 𝐶 = 512 and used random input sequences of 1-byte characters.
Table 4.3 compares the performance of the LCS benchmark with diﬀerent scheduling policies on ITO-A
with 16 nodes. Continuation stealing with greedy join achieved the best performance; it was an order ofmagnitude faster than continuation stealing with stalling join and two orders of magnitude faster than childstealing. Stalling join is much slower than greedy join because the lack of thread migration at joins led toa larger load imbalance. In particular, execution delay of futures near the root was the bottleneck. In childstealing, we found that almost all tasks were executed by the main worker because tied tasks were nevermigrated2 .
2 Load balancing did not work in child stealing. Most tasks were popped from the queue and suspended immediately before being51

4. Scalable Continuation Stealing on Distributed Memory
Next, we validate the performance of continuation stealing (greedy join) by comparing it against thetheoretical ideal performance. Figure 4.12 shows the experimental results. The ﬁlled area for each 𝑁 is
enclosed by the lines representing the lower bound 𝑇𝑃 ≥ max(𝑇1 /𝑃, 𝑇∞ ) and the upper bound of the greedyscheduling theorem 𝑇𝑃 ≤ 𝑇1 /𝑃 + 𝑇∞ (see Section 2.2.2). Note that the greedy-scheduling theorem assumes
an ideal greedy scheduler with no run-time overhead; the performance of real-world schedulers can be out of
this bound. Suppose that the execution time for each leaf block is 𝑇𝑐 , the work and span can be calculated as
𝑇1 = (𝑁/𝐶) 2𝑇𝑐 and 𝑇∞ = (2𝑁/𝐶−1)𝑇𝑐 . The value of 𝑇𝑐 was 0.340 ms for ITO-A and 0.872 ms for Wisteria-O,
which was measured by serial execution for 𝑁 = 216 . The results show that most of the parallel execution times
were within these bounds, which suggests that almost no tasks were unnecessarily blocked by the schedulerof continuation stealing (greedy join). Despite a few exceptions, continuation stealing successfully scaled to
9,126 cores on ITO-A and 27,648 cores on Wisteria-O. We could not see any performance improvements
on Wisteria-O with 110,592 cores, which needs further performance investigation. More sophisticatedscheduling policies for futures [116] might help to further improve performance, but this is beyond the scopeof this dissertation.

4.6. Summary
In this chapter, we have demonstrated that distributed continuation stealing can scale to as many as 110,592
cores. In-depth analysis revealed that, even though continuation stealing incurred a slightly longer steal latency(less than 20% overhead), it achieved better overall performance than child stealing in several benchmarks byresolving task join primitives more eﬃciently. Particularly when the synchronization pattern is complicated(e.g., futures in LCS), the performance penalty due to the lack of thread migration was substantial.
To focus on multithreading performance, this chapter has not included applications that use global memory; data are only exchanged via arguments or return values of tasks. The next chapter will introduce aneﬃcient software cache implementation for global memory access, and then we will provide comprehensive
performance evaluation of Itoyori by using real-world applications with global memory access in Chapter 7.

stolen, because steal operations are much slower than local pop and suspend operations.

52

5. Leveraging Locality of Global Memory Accesses
via Software CachingThis chapter introduces the implementation of Itoyori’s PGAS system, particularly the software cache implementation for checkout/checkin APIs (Section 3.3). The software cache is intended to exploit locality ofreference from close tasks in fork-join parallelism that are scheduled on the same process (Section 2.2.4). The
content of this chapter is based on our published paper [5].

5.1. Overview
The challenge in distributed-memory fork-join parallelism is two-fold: inter-node dynamic load balancing
and remote memory access. The previous chapter (Chapter 4) demonstrated great scalability of Itoyori’s
distributed work stealing, reaching to more than one hundred thousands of cores [4]. A remaining issue is
how to eﬃciently handle remote memory access in the PGAS model. The majority of existing PGAS systems
are designed for the SPMD-style execution, wherein the programmer is responsible of mapping computations
to nodes. To the best of our knowledge, only a few PGAS systems [58], [59] have been designed for global
fork-join parallelism (Section 2.2.5), in which tasks are automatically load balanced by the runtime system
across node boundaries.
It is challenging, however, to reconcile the PGAS model and the global fork-join model. The PGAS model
is about distinguishing between local and global data to optimize data movement, while the very point of the
global fork-join model is that tasks can move across nodes for load balancing. For instance, even if two tasks
that access the same data are likely to be executed on the same node, aggregating communication for themis diﬃcult for programmers because they could potentially run on diﬀerent nodes. Consequently, each task
communicates independently for the data it uses, resulting in ﬁne-grained and redundant communication.
A viable solution to this issue is to incorporate a software cache within the PGAS runtime. When a processor
executes a set of parallel tasks that access adjacent or overlapping memory regions, their memory accesses arelikely to be cached in the local memory, thereby reducing redundant communication. We consider this approachis eﬀective because (1) most tasks usually do not migrate when scheduled by work stealing if parallelism is
suﬃcient [9], [26], [90], and (2) tasks that are close in the DAG often access the same data [27], [117] (seealso Section 2.2.4). This approach can be seen as a compromise between DSM and PGAS (Section 2.3), as
it employs a software cache while still requiring explicit APIs. Note that this idea itself is not novel. For
example, PGAS systems such as MuPC [118], Chapel [119], CLaMPI [36], GAM [120], and Falcon [121]have implemented a software cache, albeit not in the context of global fork-join parallelism.
Unlike previous approaches that integrated the PGAS and global fork-join model [58], [59], Itoyori wasdesigned with software caching in mind, which diﬀerentiated its APIs and implementation from theirs.
Speciﬁcally, checkout/checkin APIs (Section 3.3) are newly designed for space-eﬃcient access to cached global
data. They can avoid creating unnecessary copies by directly exposing the runtime-managed cache memory
to the user, which is impossible in the conventional GET/PUT APIs. Programmability is also improved by53

Local Memory
GET

Address Space

Address Space

5. Leveraging Locality of Global Memory Accesses via Software Caching

Communication(if needed)
PUT

Communication(if needed)
Checkout

GET
Memory Copy

Checkin(always)
PUT
Global Runtime

User

Memory Cache Memory(a) GET/PUT with a software cache.

GlobalLocal

MemoryMemory(b) Checkout/Checkin.

Figure 5.1.: Cache management for diﬀerent PGAS APIs.
supporting unique virtual addresses for global memory, as detailed in Section 5.2. The checkout/checkinAPIs are enabled by dynamic manipulation for virtual-physical memory mappings, which we will cover in
Section 5.3.
In addition, Itoyori’s cache implementation adheres to the work-ﬁrst principle (Section 2.2.2). This principle
suggests that for eﬃcient work-stealing scheduler implementations, the overhead at each fork/join should be
moved to the less frequent work-stealing events. As such, we aim to delay costly coherence actions (e.g.,cache invalidation and write-back) until work-stealing events occur. To achieve this property, we designed aneﬃcient cache coherence protocol that leverages RDMA (Section 5.4).

5.2. Rationale of Checkout/Checkin APIs
In this section, we explain why we are introducing new checkout/checkin APIs. Previous approaches added asoftware caching layer without changing the conventional GET/PUT APIs [36], [118]–[121]. However, they
have shortcomings in terms of both eﬃciency and programmability.
First, GET/PUT APIs introduce unnecessary data copying between the runtime cache and user memory,
given their semantics of memory copying between the global and local memory. This issue is depicted in
Figure 5.1a. For instance, even if a GET request results in a cache hit and thus omitting communication, thedata still needs to be copied from the runtime cache to the user memory. In contrast, checkout/checkin APIs
simply require a uniﬁed global address. This allows the direct exposure of the runtime cache memory to theuser without any redundant copying, as illustrated in Figure 5.1b. In addition, even if a process checks out thesame or overlapping regions at the same time, it does not lead to any space overhead. With this API design, acache can be shared among multiple processes within the same node in the future, although this has not been
implemented yet.
Similarly, GET/PUT APIs incur unnecessary data copying, even when the requested global region is local.
Although many PGAS libraries oﬀer global-to-local pointer conversion for portions of global memory thatare known to be local [70], [72]–[75], this feature assumes the SPMD model with no inter-node dynamic loadbalancing. In scenarios with global task parallelism, programmers would be required to insert a conditional
branch for each global memory access to check if the current process owns the data. This is particularly
diﬃcult when the access region spans both local and remote memory. Arguably, checkout/checkin APIs oﬀer

54

5. Leveraging Locality of Global Memory Accesses via Software Caching

Checkout/CheckinProcess 1

Global View andMemory Mappings

Cache Blocks
RMA
Process 1

Process 2

Process 3

Process 4

Home Blocks
MemoryDistribution
Figure 5.2.: Overview of memory management in Itoyori.
a much simpler and more straightforward interface, as they can be consistently used for both global-to-local
pointer conversion and remote data access without any copying overhead.
From a programmability standpoint, checkout/checkin APIs allow a broader range of data types in line with
C++ semantics. Since GET/PUT calls essentially act as a memcpy() function, only “trivially copyable” objects
can be stored in global memory, as also noted in the UPC++ documentation [122]. This implies that certain
data types, including vector containers, cannot be made global. This limitation is an actual issue in ExaFMM(Section 7.5). In contrast, checkout/checkin APIs neither create copies nor change the virtual addresses ofobjects, which allows for nontrivially copyable types. Admittedly, data are physically copied across nodes as
raw bytes, but from the perspective of programmers, this is not considered a copy operation because virtual
addresses are never changed (similar to how hardware caches work). In addition, checkout/checkin APIs
simplify array indexing by consistently preserving the virtual addresses across their calls.

5.3. Implementation of the PGAS Library and Software Cache
This section explains the design and implementation of the cache system of Itoyori. How to integrate it with
distributed work stealing is explained in Section 5.4.

5.3.1. Overview
To realize uniﬁed global addresses for the checkout/checkin APIs, Itoyori preserves the same virtual address
space as a global view for each process and uses its addresses as global addresses. To limit the physical memory
usage, physical pages are dynamically mapped to/unmapped from the global view on demand. Figure 5.2
illustrates the virtual-to-physical memory mappings. The memory mappings are updated at the granularity ofmemory blocks of ﬁxed size (a multiple of the system page size). The upper part of the ﬁgure shows the global
view and memory mappings to the two types of physical memory blocks: home blocks and cache blocks.
Home blocks are the local portions of global memory and are mapped directly to the corresponding virtual
memory addresses. Cache blocks are used to store local copies of remote memory and mapped to the global
view on demand. Cache blocks can be remapped to other locations when the memory is not being checked55

5. Leveraging Locality of Global Memory Accesses via Software Cachingout. The number of cache blocks is ﬁxed in the current implementation and can be conﬁgured by the user at
program startup.
Physical memory blocks are allocated as POSIX shared memory with the shm open() call. POSIX sharedmemory can be used to dynamically change the virtual-to-physical memory mappings with the mmap() call.
In addition, this enables sharing of physical memory blocks among intra-node processes, even though Itoyori
spawns one process per core. Home blocks are shared among intra-node processes when created, so that they
can be directly mapped to each process’s global view. Therefore, processes can directly access the home blocks
owned by other processes within the same node. Cache blocks are not shared in the current implementation,so each process only has private caches.

5.3.2. Memory Allocation
As explained in Section 3.3.1, Itoyori supports memory allocation with collective and noncollective distribution
policies. This section describes how virtual and physical memory are managed internally in the Itoyori runtime
system.
In collective policies, home blocks are assigned to processes (called owners) by following a certain distribution policy. For example, the bottom part of Figure 5.2 shows the block-cyclic distribution. At the allocation
time, the same virtual address space of the requested memory size is newly preserved in all processes. At thistime, the virtual address space is attached no physical memory (mmapped with no access privilege). Then,physical home blocks are allocated in their owner processes and exposed to other processes for remote memory
access (calling MPI Win create()).
In contrast to collective policies, Itoyori pre-allocates a suﬃciently large virtual address space for noncollective allocation at program startup. This virtual address space is divided evenly among all processes, andeach process allocates memory from its local portion. We can either preallocate physical memory of ﬁxed sizein each process at program startup (using MPI Win create()) or dynamically attach physical memory as theheap size grows (using MPI Win create dynamic() and MPI Win attach()) for noncollective allocation.

5.3.3. Global View Management
Checkout/Checkin Implementation

The primary task of the checkout call is to fetch remote data to cache blocks and map them to the global view.
If cache blocks are already mapped and have up-to-date data (i.e., not invalidated by the synchronization callsin Section 5.3.4), it can skip communication and immediately return.
Figure 5.3 shows an implementation of the checkout/checkin APIs. The MemBlock structure (line 67–73)
is allocated for each physical memory block. Both the Checkout and Checkin functions iterate over thevirtual memory blocks that overlap with the requested region [addr, addr + size) and operates on each block(line 76–90 and line 98–103). mbID is a unique ID for each virtual memory block, which is calculated by
dividing the starting virtual address by the ﬁxed size of memory blocks (MBSize).
The GetMemBlock function (line 77 and line 99) queries the physical memory block associated with mbID.
This translation involves two separate ﬁxed-size hash tables for home and cache blocks, respectively. If thehash table already has an entry for the given ID, the handler for the associated physical memory block (mbof type MemBlock) is returned; otherwise, the ID is associated with a free memory block and its handler isreturned. If no free memory block is found, an existing mapping entry is evicted based on the least recently
used (LRU) policy. The LRU priority is managed with a doubly-linked LRU list. When a memory block is56

5. Leveraging Locality of Global Memory Accesses via Software Caching

67 Struct MemBlocktype :: HomeBlock — CacheBlock
physMem :: handler for physical memory allocated for this block.
70
addr :: virtual address to which this block should be mapped.
71
mappedAddr :: virtual address to which this block is now mapped.
72
validRegions :: set of up-to-date memory regions within this block.
73
refCount :: reference count for this block (default: 0).
74 Function Checkout(addr, size, mode)75
memBlocksToMap ← {}76
for mbID ← ⌊ addr/MBSize⌋ to ⌈ (addr+size)/MBSize⌉ − 1 do77
mb ← GetMemBlock(mbID)78
mb.addr ← mbID × MBSize
79
if mb.type = CacheBlock then80
reqRegion ← [mb.addr, mb.addr + MBSize) ∩ [addr, addr + size)81
if mode = Write then82
mb.validRegions ← mb.validRegions ∪ reqRegion
83
else if reqRegion ⊄ mb.validRegions then84
paddedRegion ← GetOverlappingSubBlocks(reqRegion)
85
fetchRegions ← {paddedRegion} \ mb.validRegions
86
BeginFetch(fetchRegions, mb.physMem)
87
mb.validRegions ← mb.validRegions ∪ fetchRegions
88
if mb.addr ≠ mb.mappedAddr then89
memBlocksToMap ← memBlocksToMap ∪ {mb}90mb.refCount ← mb.refCount + 1
91
for mb ∈ memBlocksToMap do
92
if mb.mappedAddr ≠ NULL then93
UnmapPhysMem(mb.mappedAddr, MBSize)
94
MapPhysMem(mb.addr, MBSize, mb.physMem)
95
mb.mappedAddr ← mb.addr
96
WaitFetchCompletion()
97 Function Checkin(addr, size, mode)98
for mbID ← ⌊ addr/MBSize⌋ to ⌈ (addr+size)/MBSize⌉ − 1 do
99
mb ← GetMemBlock(mbID)
100
if mode ≠ Read and mb.type = CacheBlock then101
reqRegion ← [mb.addr, mb.addr + MBSize) ∩ [addr, addr + size)
102
RegisterDirtyRegion(mb, reqRegion)
103
mb.refCount ← mb.refCount − 1
68
69

Figure 5.3.: Implementation of checkout/checkin operations.

57

5. Leveraging Locality of Global Memory Accesses via Software Cachingqueried (GetMemBlock()), its LRU entry is moved to the tail of the LRU list. Upon eviction, the LRU list istraversed from the head to the tail until an evictable memory block is found. If no evictable block is found, thecheckout function raises a too-much-checkout exception.
A memory block is evictable if it is not dirty (see Section 5.3.4) and its reference count is zero. The
reference count is incremented on the checkout call (line 90) and decremented on the checkin call (line 103).
This ensures that physical memory is present while the region is being checked out1 . We do the same for home
blocks, the reason for which will be explained in Section 103.
After getting a cache block, we make sure that the requested data are up-to-date (line 79–87). In order to manage which parts of a block are up-to-date, each cache block maintains a set of valid regions (mb.validRegions).
This is currently implemented as a linked list of byte-granularity intervals, although a bitmap would be anotheroption. If the access mode is write-only, the exact region requested by the user (reqRegion) is added to thevalid regions without fetching remote data (line 82). Otherwise, we check if the requested region is up-to-date(line 83), and if not, the remote data are fetched. The remote fetch is performed at the sub-block granularity
to exploit spatial data locality. That is, each memory block is logically divided into one or more equal-sized
sub-blocks, and the sub-blocks that overlap with the requested region are fetched at once (line 84). Notethat the already valid regions should not be fetched (line 85), so as not to overwrite the dirty data. Then,
nonblocking communication (using MPI Get()) is issued to fetch the selected regions fetchRegions from thedata owner (line 86). The completion of communication is awaited at the end of the Checkout function (using
MPI Win flush all() at line 96).
Virtual memory mappings are updated after starting nonblocking communication for all blocks, in order tohide the overhead of the mmap() system call. In Figure 5.3, mb.mappedAddr is the virtual address that the block
is currently mapped to, and mb.addr is the address that it needs to be mapped to. If they are diﬀerent (line 88),
its memory mapping needs to be updated. Such memory blocks are added to a list memBlocksToMap, andafter all communication requests are issued, their memory mappings are updated with mmap() (line 91–95).
Therefore, virtual addresses of cache blocks can change during communication, but this is not a problem
because they are assigned diﬀerent virtual addresses for communication in advance.
The Checkin function is responsible for managing dirty data in each cache block, which is registered by theRegisterDirtyRegion function if not read-only (at line 102). The management of dirty data depends on thecache coherence protocol explained in Section 5.3.4.
Saving the Number of Memory Mapping Entries

Itoyori potentially creates many memory mappings with the mmap() system call, but unfortunately, Linux has
a limit to the number of memory mapping entries. In our experimental environment (Section 7.1), we cancreate only 65530 mapping entries per process2 , which is problematic in practice. Therefore, we explicitly
unmap3 the previous mapping when the mapping changes (line 93 in Figure 5.3), although we do not need to
do this if there is no such limitation.
Nevertheless, the total cache size in Itoyori is restricted by this limitation. A memory mapping entry is
counted for each contiguous region of virtual memory that is also contiguous in physical memory. For 𝑁 cache
blocks, we would need 2𝑁 + 1 entries in the worst case (i.e., when the mappings are interleaved), because we
1 The reference count cannot be less than zero, as long as the checkout/checkin APIs are called in pairs (see the API usage in

Section 3.3.2).
2 checked with the “sysctl vm.max map count” command.
3 mmap() with no access privilege (PROT NONE) is used to preserve the virtual addresses while unmapping physical memory.

munmap() call is not used for this purpose.

58

The

5. Leveraging Locality of Global Memory Accesses via Software Cachingalso need to preserve the addresses to which no block is mapped. As the minimum block size is 64 KB in ourenvironment, the maximum cache size is approximately 65530/2 × 64 KB ∼ 2 GB for each process.
Even for home blocks, this limitation is problematic for certain memory distribution policies with interleaved
memory mappings (e.g., block-cyclic distribution). As a workaround, we limit the number of home blocks
that can be simultaneously mapped for each process. This is why home blocks are managed similarly to cache
blocks, using a hash table and reference counts in Figure 5.3. This means that home blocks are also subject
to eviction and no longer statically mapped to the global view. Because of this reason, Itoyori requires all
global memory accesses go through the checkout/checkin calls, even if the requested region is known to belocal. Note that we can skip dynamic home block management for block distribution, in which consumptionof memory mapping entries is small.

5.3.4. Cache Coherence Protocol
Itoyori employs a relaxed memory consistency model of SC-for-DRF [65], as mentioned in Section 3.3.3. To
follow the synchronization order in Itoyori, each process performs coherence actions for its local cache blocks.
A release fence ensures that all dirty data in the local cache are written to their homes, and an acquire fence
self-invalidates all local caches (by clearing validRegions in Figure 5.3) so that successive checkout operations
will fetch the latest data from their homes. In this way, Itoyori ensures the order between the write events
before a release fence and the read events after the associated acquire fences. Although one might feel that thiscoherence protocol is too naive, similar approaches are taken by GPU’s hardware caches [123] and Chapel’s
software cache [119] because of its simplicity.
In this dissertation, we consider two approaches to handling dirty data (RegisterDirtyRegion() at line 102
in Figure 5.3): the write-through and write-back policies. With the write-through policy, the dirty data are
written to their homes immediately on each checkin call, without remembering the dirty regions. On the other
hand, with the write-back policy, we delay ﬂushing dirty data until the next release fence by remembering thedirty regions. The dirty regions are managed on a per-block basis and are maintained in the same way as thevalid regions (validRegions) as a linked list of memory regions. When a release fence is executed, all dirty
regions are written back to their homes in byte granularity. As long as a cache block has any dirty region, theblock is not evictable. If all cache blocks are not evictable when a free cache block is needed, then the systemwrites back all dirty data and retries the eviction procedure.

5.4. Integration with Work Stealing
This section explains how Itoyori integrates the cache system with work stealing, with an aim to to follow the
work-ﬁrst principle [9] by delaying costly coherence actions until work stealing occurs.

5.4.1. Release/Acquire Fences in Work Stealing
As Itoyori’s threads can be dynamically migrated to other processes at fork-join calls, release/acquire fences
are inserted to fork-join points. Figure 5.4 shows possible program points to insert release/acquire fences.
Since the modiﬁcations made before the fork must be read by the child and the continuation of the parent, we
insert release/acquire fences accordingly. Similarly, the process that executes the continuation of the join must
read the modiﬁcations made by the child and the parent before the join.

59

5. Leveraging Locality of Global Memory Accesses via Software Caching

Release #1 Acquire #2

Release #2 Acquire #1

Parent Thread
Fork

Join

Child Thread
Acquire #3

Release #3

Figure 5.4.: Possible release/acquire fences in fork-join calls.
104 Struct ProcessLocalData

myProcID :: ID (rank) of the local process.
currentEpoch :: current epoch of the release operation.
107
requestEpoch :: epoch of the data requested to release by others.
108 Struct ReleaseHandler
109
procID :: ID (rank) of the process who owns the data to be released.
110
epoch :: epoch of the data to be released.
111 Function ReleaseLazy() :: ReleaseHandler
112
if all cache blocks are clean then return Unneeded
113
else return (myProcID, currentEpoch + 1)
114 Function Acquire(handler :: ReleaseHandler)
115
if handler ≠ Unneeded then116
while Get(currentEpoch at handler.procID) < handler.epoch do
117
if ﬁrst time then118
atomic op at handler.procID do
119
requestEpoch ← max(requestEpoch, handler.epoch)
120
Invalidate all cache blocks.
121 Function DoReleaseIfRequested()
122
if currentEpoch < requestEpoch then123
Write back all dirty data to their homes.
124
currentEpoch ← currentEpoch + 1
105
106

Figure 5.5.: Implementation of lazy execution for release fences.
Obviously, this naive approach is not eﬃcient for ﬁne-grained parallelism, but if we assume certain scheduling policies, we can reduce the number of fences. As mentioned earlier (Chapter 4), the work-stealing schedulerof Itoyori follows the work-ﬁrst policy. As the child thread is immediately executed after the fork, Acquire #3
in Figure 5.4 can always be skipped. In addition, as long as the parent thread is not stolen, Release #2, #3
and Acquire #1, #2 can be skipped, because the child thread can be treated as a serialized function call [9].
Conversely, if the parent thread is stolen by another process, all of these fences are executed. Release #1 is the
only fence that is nontrivial to skip, because we do not know in advance if the parent thread will be stolen ornot.

5.4.2. Lazy Execution of Release Fences
The release fence before forking (Release #1) is, unlike the fences that are conditionally executed only whenwork stealing happens, performance critical for ﬁne-grained parallelism. This is because fork-join calls areusually much more frequent than work stealing events (cf. the work-ﬁrst principle [9]). That is, the more

60

5. Leveraging Locality of Global Memory Accesses via Software Cachingﬁne-grained the threads are, the more release fences will be executed.
Therefore, we consider delaying the execution of Release #1 until the parent thread is stolen. This would
require the thief to notify the victim of the steal event and wait for the release to complete. However, naive
implementations would cause frequent interruptions on the victim (e.g., by active messages), which candiminish the beneﬁts of RDMA-based asynchronous work stealing.
Following the work-ﬁrst principle, we designed an algorithm that can minimize the victim’s overhead. In
our algorithm, the thief sends a release request to the victim and the victim polls the requests. Each polling
operation can be performed quickly because the check does not involve a communication call, since it only reads
local variables that may be modiﬁed by remote processes via MPI-3 RMA (assuming the MPI WIN UNIFIED
model).
Figure 5.5 shows our implementation. The ReleaseLazy function (line 111–113) is the function to be
executed at Release #1. It returns a release handler (ReleaseHandler at line 108–110), which is later passed
to the thief and used to request a write-back for the dirty data at this point. A release handler is a pair of theprocess ID (MPI rank) and an epoch. An epoch is managed by each process (currentEpoch) and incremented
at each write-back operation by the process. If the local cache is dirty, currentEpoch + 1 is returned as anepoch for the release handler (line 113). This indicates that the next write-back operation must be completed
by this process to ensure the synchronization order. If the cache is clean, then no write-back request is needed
and Unneeded is returned as a release handler. Release handlers are then passed to the corresponding acquire
fences (Acquire #2) by value.
The Acquire fence function (line 114–120) is called only when the parent thread is stolen. If the handler
is not Unneeded, it ﬁrst fetches the current epoch of the releaser to check if the next write-back operation
has already been performed (line 116). If it is still smaller than the required epoch (handler.epoch), thena write-back request is sent to the releaser only once (line 118–119). Our insight is that, even if multiple
acquirers simultaneously send write-back requests to the same releaser, only the maximum epoch among themis suﬃcient. Therefore, we use a remote atomic operation to set the maximum epoch at the releaser’s memory4 .
The acquirer then waits until the remote epoch reaches the required epoch by repeatedly getting the remote
epoch (using MPI Get()).
To periodically check the update on the requested epoch, the polling function (DoReleaseIfRequested
at line 121–124) is inserted to each fork and join call. If the requested epoch is greater than the current
epoch, the process writes back all dirty data and increments the current epoch, so that the acquirers can breakthe loop. Note that long-running tasks can delay the execution of the polling function for a long time inthis implementation. Another approach would be to call DoReleaseIfRequested in a dedicated kernel-level
thread, but this has not been implemented yet.

5.5. Summary
In this chapter, we discussed the beneﬁts of checkout/checkin APIs and their detailed implementation with
software caching. The interfaces of checkout/checkin enable avoiding copy semantics, thereby beneﬁting
both performance and programmability. This is enabled by their implementation that utilizes virtual memory
mappings. Furthermore, we explored techniques to eﬃciently cache data across multiple tasks scheduled
to the same process by considering interactions with the work-stealing scheduler. The actual performance
4 The maximum value can be set by using the MPI Fetch and op() function with the MPI MAX operation, but this operation is usually

not oﬄoaded to RDMA. Instead, we emulate this operation using the MPI Compare and swap() function with a loop.

61

5. Leveraging Locality of Global Memory Accesses via Software Caching
improvements gained by software caching is later evaluated in Chapter 7.

62

6. Locality-Aware Task Scheduling with Almost
Deterministic Work Stealing (ADWS)
Prior to this chapter, Chapter 4 discussed fundamental runtime techniques for implementing distributed
continuation stealing, and Chapter 5 introduced a software cache to leverage locality of global memory access.
Nevertheless, we have assumed randomized work stealing as the scheduling algorithm, which completely
ignores the memory hierarchy. This chapter attempts to leverage data locality in fork-join parallelism in deep
memory hierarchies by introducing a new scheduling algorithm, called ADWS. We have introduced the basic
idea of ADWS in our conference paper [1], but the content of this paper is based on our journal paper [3],which provides more reﬁned explanation for ADWS and introduces multi-level scheduling.

6.1. Overview
Fork-join parallelism enables programmers to write high-level code to express many parallel algorithms,
including parallel divide-and-conquer algorithms. This programming style often naturally leads to gooddata locality, because the working set is divided into smaller ones as a problem is recursively decomposed. A
representative set of algorithms that use this data locality property is known as cache-oblivious algorithms [31],[32]. While these algorithms are oblivious to speciﬁc cache hierarchies and sizes, they are proved to haveasymptotically optimal bounds on the number of cache misses. The key to cache-oblivious algorithms is
to organize data locality in a hierarchical way so that the data locality matches any cache conﬁguration.
Thus far, and many parallel variants of cache-oblivious algorithms have been developed by using fork-joinparallelism [117], [124].
Nevertheless, it is challenging to exploit the data locality of fork-join parallelism on hierarchical caches(Section 2.1.1). Work stealing is known to eﬃciently utilize private caches (Section 2.2.4) but not hierarchicalcaches. Even on a processor with a single shared cache and multiple private caches, it has been reported [50]that work stealing is not optimal for data sharing on a shared cache, and that parallel depth-ﬁrst (PDF)
scheduling [51] outperforms work stealing. For deeper and more complicated cache hierarchies, however,neither work stealing nor PDF scheduling is optimal; accordingly, for arbitrary cache hierarchies, spacebounded schedulers [117], [125], [126] have been proposed.
Another aspect of the data locality of fork-join parallelism is that, when the same or similar computations
are repeatedly executed on the same working set, parallel tasks at the same position in the task hierarchy
are likely to access the same data across multiple iterations [27]. This ﬁnding is based on the observationthat many algorithms exhibit almost the same memory access pattern for the same array across consecutive
iterations. This direction has also been investigated: constrained work stealing [127] and LAWS [128], [129]improve data locality by scheduling tasks somewhat deterministically, but they are not designed for arbitrary
cache hierarchies. Space-bounded schedulers, on the other hand, are designed for arbitrary cache hierarchies
but do not account for iterative data locality. To the best of our knowledge, no scheduler has been designed forboth iterative computations and arbitrary cache hierarchies at the same time.

63

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Hence, to fulﬁll both of these requirements, we introduce a simpler and more straightforward approach,called Almost Deterministic Work Stealing (ADWS). ADWS ﬁrst deterministically maps tasks to cores so thatthe task hierarchy matches the cache hierarchy, which enhances data locality on hierarchical caches. This
deterministic task mapping also enhances data locality for iterative computations, because, at every iteration,
data at the same location are likely to be accessed by the same core. To execute ADWS, work hints must be
given when new tasks are forked. A hint is the ratio of the computational work in the forked task to the workin the continuation of the running task until the forked task is joined. Note that specifying the work hints is
not mandatory for programmers. The runtime system can automatically infer them (e.g., ratio of 1:1), while
explicitly providing accurate work hints can improve the performance. Even if the work hints are wrong,
ADWS performs dynamic load balancing to ﬁx load imbalances derived from the initial deterministic task
mapping, which is why ADWS stands for almost deterministic work stealing.
Then, we further extend ADWS to achieve better utilization of shared caches. The problem with the baseline
design of ADWS is that it does not facilitate shared cache reuse when the overall working set size is muchlarger than the shared cache sizes. Accordingly, when a set of cores sharing a cache receives a task whose
working set size is much larger than the shared cache, parallel execution of its descendants may result in a large
number of capacity misses. In other words, ADWS may cause many more shared cache misses that would
occur in serial execution, because cores sharing a cache may execute almost independent parts of a problem,so that almost no cache reuse is expected.
Such ineﬃciency in the use of shared caches is not speciﬁc to ADWS but also applies to work stealing andits variants. To date, many approaches have addressed this issue, including CATS [130] and space-boundedschedulers [117], [125], [126], to name a few. By borrowing the underlying concept of those approaches, we
introduce a scheduling framework called multi-level scheduling, which generalizes the concept of two-level
scheduling in the previous literature [128]–[132]. Whereas the previous concept of two-level schedulingassumed speciﬁc cache conﬁgurations and speciﬁc scheduling strategies, multi-level scheduling generalizes
the concept for any level of a cache hierarchy, with arbitrary schedulers for each cache level. The idea ofmulti-level scheduling is that, for each shared cache, the cores that share it simultaneously execute only one
task (and its descendants) whose working set size is small enough to ﬁt into the shared cache, while leaving
other tasks unscheduled. We assume that estimated space usage for each task is given by the programmer as
an additional hint. Once a task is assigned to a shared cache, its descendants are scheduled recursively to thechildren of the cache. For each cache level, we can arbitrarily choose the scheduling policy to map tasks fromthe parent cache to the children. By using the framework of multi-level scheduling, we have straightforwardly
devised an extension of ADWS, called multi-level ADWS, in which the baseline version of ADWS (called
single-level ADWS here) is applied to each cache level. Other variants of multi-level schedulers can also
be considered by applying diﬀerent scheduling strategies to each cache level: for example, we can consider
multi-level work stealing, in which traditional random work stealing is applied hierarchically.
To empirically study the performance of our approach, we conducted a comprehensive performance analysis
using seven benchmarks on a two-socket Cascade Lake machine (shared memory). In this analysis, we
compared ﬁve schedulers: single-level WS (traditional random work stealing), single-level ADWS, multi-level
WS, multi-level ADWS, and a space-bounded scheduler (a port of the implementation by Simhadri et al. [133],[134]). The results show that multi-level ADWS outperformed the other schedulers on highly memory-boundbenchmarks over a wide range of working set sizes. Notably, for decision tree construction with a large realworld dataset having a size of about 2 GB, both single- and multi-level ADWS outperformed Cilk Plus [103], awidely used tasking runtime with single-level WS, by 15% and 60%, respectively. These results also conﬁrm
the advantage of multi-level ADWS over single-level ADWS. Moreover, multi-level ADWS outperformed

64

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

125 Function BuildDecisionTree(rows)
126
127
128
129
130
131
132
133
134
135
136
137

foreach attr ∈ attributes do
split ← ComputeBestSplit(rows, attr)
if split is better than bestSplit thenbestSplit ← split
if Splitting at bestSplit is beneﬁcial then(rowsL, rowsR) ← Partition(rows, bestSplit)
spawn nodeL ← BuildDecisionTree(rowsL)
spawn nodeR ← BuildDecisionTree(rowsR)
sync
return InnerNode(nodeL, nodeR)
else
return LeafNode(rows)
Figure 6.1.: Algorithm for parallel decision tree construction.

multi-level WS by 18%, which conﬁrms the beneﬁt of exploiting data locality for iterative computations.
Other experiments demonstrated that ADWS is tolerant to load imbalances even when work hints are incorrect,
and that NUMA local allocation policy can be exploited by deterministic scheduling of ADWS.
Finally, we also implemented ADWS in Itoyori (only single-level ADWS). The distributed-memory implementation can be straightforwardly ported from the shared-memory ADWS implementation, because of
ﬂexible multithreading capabilities discussed in Chapter 4. Nevertheless, there is a performance issue speciﬁc
to distributed memory, and we describe how we have addressed this issue in Section 6.7.

6.2. BackgroundIn this section, we ﬁrst introduce a motivating example for fork-join parallelism in Section 6.2.1. Then, we
explain notation and terminology used for our explanation in this chapter (Section 6.2.2), and then discuss two
types of data locality in fork-join parallelism.

6.2.1. Motivating Example: Decision Tree Construction
A decision tree can be constructed by a simple divide-and-conquer algorithm, that can be straightforwardly
parallelized by nested fork-join constructs, as studied in [104]. This paper assumes the CART [135] algorithm
for binary classiﬁcation with continuous-valued attributes.
Figure 6.1 shows a simpliﬁed algorithm for decision tree construction. It receives training data (rows) as
input, each row of which consists of a class (0 or 1) and multiple attributes with continuous values. The
resulting decision tree is then used to predict the class from given attributes. At line 131, the rows are
partitioned into two parts on the basis of bestSplit, which speciﬁes which column (attribute) is used to split therows, and its threshold. Then, the partitioned rows (rowsL and rowsR) are used to recursively construct two
subtrees, which can be computed in parallel (line 132–133). These two parallel tasks are created by the spawn
keyword, and the wait for their completion is indicated by the sync keyword (Cilk-style synchronization).
Before partitioning, bestSplit is chosen for the split with the lowest value of the Gini impurity [135], where
this choice involves consecutive parallel computations (line 126–129). The best split is ﬁrst determinedfor each attribute (split at line 127) and bestSplit is then chosen from among all the attributes. This paper65

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

Iterative data locality

Task

Task Group
Hierarchical data locality

Figure 6.2.: Simpliﬁed model for fork-join parallelism.

Figure 6.3.: Example of a DAG for decision tree construction.

assumes that the best split for each attribute is determined by constructing histograms [136], [137], as eﬃcient
implementations such as LightGBM [138] do, rather than by sorting the rows by each attribute. As a result,
the ComputeBestSplit function at line 127 can be written as a simple, ﬂat parallel loop (or parallel reduction)
to construct a histogram. The Partition function can also be parallelized through double buﬀering as in
Quicksort.

6.2.2. Notation and Terminology
This chapter assumes computations with fork-join parallelism (Section 2.2.1). Here, we deﬁne a task groupas a group of tasks that are joined at once. For example, the tasks in the task group construct (Figure 2.4b)
constitute a task group. Similarly, the tasks spawned by a parallel invoke() call constitute a task group too.
In this chapter, without a loss of generality, we deal with a simpliﬁed model of fork-join parallelism for
simplicity of explanation. Note that our ADWS implementation is not limited to the simpliﬁed model; it candeal with fork-join parallelism in general. In the simpliﬁed model, all tasks in a task group are spawned at once,
and then the parent task immediately issues a join call for them all. Figure 6.2 illustrates the simpliﬁed modelof the fork-join model. In each task group, all tasks are spawned and joined at once. A task can sequentiallycreate multiple task groups. This model can be expressed with the task group notation by always spawning
the last computation as a task (e.g., D() in Figure 2.4b is also spawned as a task). parallel invoke() always
follows this simpliﬁed model.
Figure 6.3 shows an example of a DAG for decision tree construction. As explained above, the procedure
is based on a two-way divide-and-conquer algorithm. In the ﬁgure, the ﬁrst (topmost) two ﬂat parallel
computations at each recursion represent the graphs for ﬁnding the best split (the ComputeBestSplit function
at line 127 in Figure 6.1), assuming only two attributes for each row. The third computation corresponds
to partitioning (the Partition function at line 131), after which two tasks are spawned and the trees arerecursively constructed. Note that this DAG is input-dependent and not known in advance: the DAG is
dynamically unfolded during execution. Although the ﬁgure shows a graph that is well-balanced for simplicity,
in reality it can be highly unbalanced, depending on the best split choice.

66

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
0

0

0

2

0

2

0

2

→
3

2

1

0

→
3

2

1

0

3

2
2

3 3

3

3

1
1

2

0
1

2 1

2

0
1

1

0
0

0

Figure 6.4.: Overview of deterministic task mapping in (single-level) ADWS with four workers. The nodes
represent tasks, and the numbers on the tasks indicate which workers will execute them. The
triangles below the nodes represent the distribution ranges over the workers to which the tasks’descendants will be distributed.

6.2.3. Two Types of Data Locality in Fork-Join Parallelism
Figure 6.3 also illustrates the data locality that resides in the task graph. Here, we introduce two types of data
locality: hierarchical data locality and iterative data locality. Hierarchical data locality speciﬁes multiple
levels of data locality in the task hierarchy: descendant tasks of the same parent are likely to access the same
subset of the data, as the working set is usually decomposed recursively into smaller ones. Iterative data locality
occurs when the same or similar parallel computations are repeatedly executed: tasks at the same positionin the graph are likely to access the same data, as many algorithms exhibit almost the same memory accesspattern across consecutive iterations. In Figure 6.3, the tasks enclosed by the dashed line have hierarchical data
locality, because the data rows are recursively partitioned into disjoint parts. In contrast, the tasks enclosedby the dotted line have iterative data locality, which typically exists in consecutive ﬂat parallel loops, becausetasks at the same position tend to access the same rows across multiple iterations. In the computations forﬁnding the best split for each attribute, a series of data is repeatedly accessed in the same order.
The task scheduler should respect both the hierarchical and iterative data locality explained above, but thisis challenging on the hierarchical caches of modern hardware. For example, work stealing is suboptimal inhierarchical caches because close tasks may be executed by workers that are distant in the cache hierarchy,
which results in many cache misses. In addition, work stealing does not respect iterative data locality becauseof its randomness [27]: cached data for previous iterations are unlikely to be reused for successive iterations.
Other schedulers that have been proposed for hierarchical caches are explained in detail in Chapter 8.

6.3. Almost Deterministic Work Stealing
This section introduces Almost Deterministic Work Stealing (ADWS), a task scheduler that is designed toaddress both hierarchical and iterative data locality on arbitrary cache hierarchies. The scheduling in ADWS isbased on deterministic task mapping but also performs dynamic load balancing to ﬁx load imbalances, which
is why we describe it as almost deterministic.

6.3.1. Deterministic Task Mapping
ADWS ﬁrst performs locality-aware task mapping by using work hints. A work hint is the ratio of amountof computational work in the forked task (and its descendants) to work in the continuation of the runningtask (and its descendants) until the forked task is joined. In the simpliﬁed model of the fork-join model,
work hints are simply the ratio of computational work for each task in a task group. Figure 6.5 shows how
to manually specify the work hints in the task group construct and parallel invoke(). In the task group67

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
task_group tg(w_all, space);tg.run([] { A(); }, w1);
tg.run([] { B(); }, w2);
tg.run([] { C(); }, w3);
5 tg.run([] { D(); }, w4);
6 tg.wait();12
34

parallel_invoke(space,[]{ A(); }, w1,[]{ B(); }, w2,
4[]{ C(); }, w3,
5[]{ D(); }, w4);12
3(a) Task group.

(b) parallel invoke().

Figure 6.5.: Speciﬁcation of work and space hints for (multi-level) ADWS.

(1)

(2)

(3)

Figure 6.6.: Classiﬁcation of tasks in a cross-worker task group. (1) Tasks passed to another worker (which
can be either cross-worker or non-cross-worker tasks). (2) Cross-worker task assigned to local
worker 𝑖. (3) Non-cross-worker tasks assigned to local worker 𝑖.
notation (Figure 6.5a), w_all is for the total work for all tasks in the task group, and w1, w2, w3, and w4 arefor the work for each task (the space parameter is used only for multi-level scheduling introduced later in
Section 6.4). The ratios do not have to be absolute values as long as the ratios of w1, . . . , w4 to w_all are
all correct. In fact, the last computation D() with work hint w4 does not have to be spawned as a task. In
parallel invoke() (Figure 6.5b), the total work w_all is not needed. Even if work hints are not provided,
the system can arbitrarily infer the work hints. For example, parallel invoke() can infer that work for eachtask is equivalent to each other (i.e., ratio 1 : 1 : 1 : 1).
Figure 6.4 illustrates the deterministic task mapping in ADWS. We assume that each worker is ﬁxed to a
speciﬁc hardware core, and workers that are close in the hardware topology are numbered adjacently. The task
tree is vertically divided so that workers that are close work on tasks that are close in the tree, thus respecting
hierarchical data locality. To exploit iterative data locality, the task mapping is generated deterministicallyby recursively dividing distribution ranges according to the ratios of work in each task group. A distribution
range is represented as a range of workers [𝑥, 𝑦), where 𝑥 and 𝑦 are real numbers (𝑥 ≤ 𝑦), and it is illustrated asa triangle below a node (task) in the ﬁgure. First, the root task has a distribution range [0.0, 𝑃), where 𝑃 is thenumber of workers, which means that the descendants of the root task should be uniformly distributed among
all workers. When a new task group is created, the current task’s distribution range is divided according to
the relative work for each child task, and each child task is assigned the corresponding subdivided distribution
range. The endpoint values of the distribution ranges can be real numbers rather than just integers; that is, aboundary between distribution ranges can be in the middle of a worker. The basic rule of scheduling is thata task with range [𝑥, 𝑦) is assigned to worker ⌊𝑥⌋ (where ⌊. . . ⌋ denotes the ﬂoor function), i.e., the rightmost
worker in the range shown in the ﬁgure. The numbers on the nodes in the ﬁgure represent the assigned workers.
To explain the detailed algorithm of ADWS, we ﬁrst classify tasks into two types. A task with range [𝑥, 𝑦) isclassiﬁed as a cross-worker task if ⌊𝑥⌋ ≠ ⌊𝑦⌋ and as a non-cross-worker task if ⌊𝑥⌋ = ⌊𝑦⌋. Cross-worker tasks

68

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
138 TG ← Task group of interest

139 Tcur ← Current task (that creates TG)
140 Function TaskGroup::Init(work)

TG.work ← work
TG.range ← Tcur .range
143 Function TaskGroup::Run(Tnew , work)
144
Add Tnew to TG.tasks
145
TG.work ← TG.work − work
146
(Tnew .range, TG.range) ← divide TG.range according to the ratio work : TG.work
147
if Tcur is a cross-worker task and Tnew belongs to another worker then148
Migrate Tnew to worker ⌊Tnew .range.𝑥⌋
149
else
150
Execute Tnew and push the continuation of current task Tcur to a local task queue
151 Function TaskGroup::Wait()
152
foreach 𝑇 ∈ TG.tasks do
153
Wait for completion of 𝑇154
if Tcur is a cross-worker task then155
Migrate Tcur to worker ⌊Tcur .range.𝑥⌋
141
142

Figure 6.7.: Algorithm for deterministic task mapping in ADWS.
have higher priority than non-cross-worker tasks, because their descendants should be distributed to multiple
workers as soon as possible. Task groups inherit the same distribution range from the task that creates them,and we classify task groups into cross-worker or non-cross-worker task groups, as well. Next, suppose thatworker 𝑖 creates a cross-worker task group; then, we further consider three kinds of tasks within the task group,
as illustrated in Figure 6.6. The basic rules to schedule these three kinds of tasks are as follows:
1. Tasks with range [𝑥, 𝑦) such that ⌊𝑥⌋ > 𝑖 are passed to worker ⌊𝑥⌋. They can be cross-worker or
non-cross-worker tasks.
2. A task with range [𝑥, 𝑦) such that ⌊𝑥⌋ = 𝑖 and ⌊𝑦⌋ > 𝑖 is immediately executed by worker 𝑖. It is across-worker task assigned to worker 𝑖, and it is guaranteed to be the only one for each cross-worker task
group.
3. Tasks with range [𝑥, 𝑦) such that ⌊𝑥⌋ = ⌊𝑦⌋ = 𝑖 are executed later by worker 𝑖. They are non-cross-worker
tasks assigned to worker 𝑖.
Finally, we explain the algorithm for deterministic task mapping, which is shown in Figure 6.7. The Init,
Run, and Wait functions correspond to those in the task group notation shown in Figure 2.4b. When a task
group is initialized, it is assigned the total work for the group (according to a programmer’s hint) and the
distribution range of the executing task (line 140–142). The child tasks in the group are created from left
to right as shown in Figure 6.6, and their distribution ranges are divided and assigned at line 145–146. The
worker ﬁrst migrates tasks of type (1) to the appropriate workers at line 147. When the worker ﬁnds a task oftype (2), it immediately executes it at line 150 and pushes the continuation of the current task (𝑇cur ) into thetask queue. The continuation left in the queue only contains tasks of type (3), which will be executed later by
the worker at line 150 in the work-ﬁrst manner.
After all the tasks of a task group are completed, the continuation of the parent task (𝑇cur ) is returned toits owner (line 155). This is because of the rule that cross-worker tasks must be executed by their owner69

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Primary
queuesMigration
queues

steal

0
1

depth

0
1

steal

migrate

depth

push

pop

push

pop

Figure 6.8.: Illustration of the task execution order for worker 𝑖 in ADWS, with the tasks numbered in that order.
𝑇0 , 𝑇1 , and 𝑇2 are cross-worker tasks assigned to worker 𝑖. Non-cross-worker tasks generated byworker 𝑖 (𝑇3 and 𝑇4 ) are pushed into the local primary queues, and those from other workers (𝑇5 ,𝑇6 , and 𝑇7 ) are migrated to the migration queues. The primary and migration queues are separated
by task depth.
(i.e., worker ⌊𝑥⌋ for cross-worker tasks with range [𝑥, 𝑦)). A cross-worker task returned to the owner isimmediately executed by the owner when it reaches a scheduling point (e.g., spawn, task completion, or
possibly preemption [2]), as it has the highest priority. Cross-worker tasks can be executed without delay
because at most one cross-worker task is executable by the same worker simultaneously. The reason is that across-worker task group can have only one child cross-worker task that is owned by the same worker (i.e., thetask of type (2) in Figure 6.6), and the parent cannot proceed until the child has ﬁnished.
Following the algorithm for deterministic task mapping, each worker executes its assigned tasks from left toright in the ﬁgures. Figure 6.8 illustrates the execution order for worker 𝑖. First, worker 𝑖 preferably executes
cross-worker tasks, which are the leftmost tasks in the ﬁgure; then, it executes the non-cross-worker tasks
that it created and those migrated from other workers. In the implementation, each worker’s local task queueis split into primary queues and migration queues, which are further divided by task depth. This separationis needed for the dynamic load balancing explained later in Section 6.3.2. Basically, a worker’s primary
queues are used for storing non-cross-worker tasks that it creates1 , while its migration queues are for passing
non-cross-worker tasks across workers. The primary queues follow last-in ﬁrst-out (LIFO) order for ordinary
push/pop operations, while the migration queues follow ﬁrst-in ﬁrst-out (FIFO) order to pop tasks that were
migrated by other workers. As the primary queues are checked before the migration queues in popping a local
task, the tasks in Figure 6.8 are executed from left to right, as the tasks 𝑇0 , . . . , 𝑇7 are numbered. This is almost
the same order as serial execution, which is assumed to be highly tuned by programmers to achieve good cacheperformance [50]. Hence, ADWS is considered to be private-cache-friendly as well as work stealing [27].
However, deterministic task mapping alone cannot always provide a good schedule for tasks. That is, it can
distribute tasks so that each worker’s work is well balanced, but it does not guarantee that workers can proceedwithout any stalls. Figure 6.9 shows an example of execution stalls caused by task dependencies. Following theleft-to-right execution policy, worker 𝑖 + 1 executes task 4 after tasks 1, 2, and 3, but worker 𝑖 cannot proceed
to the next task group until task 4 is completed. Thus, worker 𝑖 is stalled until worker 𝑖 + 1 executes task 4,
and subsequently, worker 𝑖 + 1 is stalled by the delayed tasks assigned to worker 𝑖. This situation happens
when task synchronization is frequent, i.e., when tasks create multiple task groups sequentially. Generally, it
is hard to provide a good schedule that does not cause any stalls due to task dependencies, because the task
1 Note that 𝑇 and 𝑇 in Figure 6.8 are in fact the continuations of 𝑇 and 𝑇 respectively, but they are abstracted as non-cross-worker
3
4
1
0

tasks, because a continuation only spawns non-cross-worker tasks for the current task group.

70

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Timeline of workers

1

4

1

2

1
2
3
4
5
6

2
5 6 3 4 5 6
3

1
2
3
4
5
6

Figure 6.9.: Example of execution stalls in the deterministic task mapping of ADWS. The numbers in the nodes(tasks) represent the execution order for each worker, and each one has the same amount of work.
The timelines show the timing of when the tasks are executed by each worker. This example showsthat even if the same amount of work is allocated to each worker, execution stalls can happen
because of task dependencies.
0

0
2
2
3

1
2

3

2

3

2

3

0

1

→

0

1

0

2

0

1

0

2

0

0

(a)

2

1
2

2

1

3

→

1

3

2

1

(b)

0

3

2

1

0

(c)

Figure 6.10.: Distribution tree for dynamic load balancing in ADWS. The nodes correspond to cross-worker
task groups, with non-cross-worker task groups omitted from Figure 6.4. Active nodes areillustrated with triangles below. The thick arrows point to the dominant nodes, which determine
the steal ranges for workers dominated by them.
graph is not known ahead of time: only the relative amount of work for the tasks in each task group is known.
Our approach to solve this problem is to combine deterministic task mapping with the dynamic load balancing
technique explained in the next section.

6.3.2. Dynamic Load Balancing
Dynamic load balancing in ADWS resolves both the execution stall problem explained above and load imbalances that dynamically appear. A load imbalance can result from the ﬂuctuation of execution times (e.g.,
OS noise, frequency scaling) and from imprecise work hints provided by programmers, as it is often hard (oreven impossible) for them to give a precise estimate for work. A naive approach to ﬁx such execution stalls
and load imbalances would be to perform random work stealing when local tasks are exhausted, but this could
substantially increase cache misses on hierarchical caches by moving data across distant caches. Moreover, asteal can incur further steals, thus collapsing the task structure made by deterministic task mapping.
Instead, our approach is to localize the range of workers for choosing a victim by using the task structure
generated by deterministic task mapping. The range of workers for work stealing (called the steal range)changes during execution, depending on the completion status of cross-worker task groups. Roughly speaking,the steal range is restricted to a set of workers that are working on the same unﬁnished cross-worker task
group. For example, in Figure 6.9, by restricting the steal range of worker 𝑖 and 𝑖 + 1 to within the task groupof interest, worker 𝑖 and 𝑖 + 1 will steal task 4 and 6 respectively, without stealing tasks outside this task group.

71

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Speciﬁcally, to determine the steal range for each worker, we use the distribution tree data structure, which
is a tree of cross-worker task groups that is created during deterministic task mapping. When a new crossworker task group is created, we create a new tree node that points to the parent node. Figure 6.10 illustrates
the distribution tree, which corresponds to the tree of cross-worker task groups of Figure 6.4, omitting thenon-cross-worker groups. Here, the tree nodes represent task groups rather than tasks. The thick arrows point
to the nodes having the current steal ranges for each worker. They are chosen from among the active nodes,
which have triangles below the nodes. A node is active if at least one of its child cross-worker tasks has alreadybeen completed. A worker 𝑖 is dominated by a node with the distribution range [𝑥, 𝑦) if ⌊𝑥⌋ ≤ 𝑖 < ⌊𝑦⌋ (worker⌊𝑦⌋ is not dominated). The current steal range of worker 𝑖 is determined by the dominant node, which is the
topmost active node that dominates worker 𝑖.
At an early stage of execution, every worker has one active node that dominates only itself (Figure 6.10a),assuming that enough parallelism is provided. As tasks are getting completed, the nodes that are close to theroot become active. In Figure 6.10b, the leaf node that dominates worker 1 has not been activated yet, but its
ancestor (with range [0, 2.𝑥)) is already active. Therefore, the current steal range of worker 1 is determined tobe that of the ancestor, which is the dominant node for worker 1. Every time a worker tries to steal a task, it
ﬁrst checks whether any ancestor has become active before performing a steal. This can be done by traversing
the distribution tree nodes from the bottom up until reaching the root, which takes time proportional to thetree depth at most. This cost will be negligible in most cases, because it is incurred only when work stealingis performed (cf. the work-ﬁrst principle [9]). Eventually, the root node becomes active (Figure 6.10c), whichis equivalent to traditional random work stealing that targets all workers.
Once the current steal range of worker 𝑖 is determined as [𝑥, 𝑦) (⌊𝑥⌋ ≤ 𝑖 < ⌊𝑦⌋), it randomly chooses worker
𝑗 from ⌊𝑥⌋ ≤ 𝑗 ≤ ⌊𝑦⌋ ( 𝑗 ≠ 𝑖) and tries stealing work from worker 𝑗. To steal the rightmost task in ﬁgures,
worker 𝑖 ﬁrst tries to steal a task from the migration queues of worker 𝑗, and if it fails, it tries to steal fromthe primary queues. As an exception, worker 𝑖 should not steal tasks from the migration queues of worker ⌊𝑥⌋
and from the primary queues of worker ⌊𝑦⌋. This rule is to avoid stealing tasks outside the dominant node’s
task group. Tasks should not be stolen from the migration queues of worker ⌊𝑥⌋, because it stores tasks thatwere migrated from outside the steal range. Similarly, tasks in the primary queues of worker ⌊𝑦⌋ are out of therange [𝑥, 𝑦). This is why we split the local task queue into primary queues and migration queues. Note that,to maintain this separation, the descendants of tasks that are migrated to migration queues are pushed into themigration queues unless stolen, and the same applies to primary queues.
Furthermore, to avoid stealing tasks from outside the dominant node, we also manage task depths andseparate primary and migration queues by task depth. A task depth is used to distinguish between tasks from
diﬀerent cross-worker task groups, and a task depth is deﬁned as the depth of cross-worker task groups in thetask hierarchy (the root depth is 0). Hence, when a non-cross-worker task group is created, the task depth is notincremented. Tasks inherit the depth of the parent task group, and tasks at depth 𝑑 are pushed into the primaryor migration queue at depth 𝑑, as illustrated in Figure 6.8. Because the descendants of a non-cross-worker task
have the same depth, they are pushed into the same queue unless stolen. For example, the descendants of 𝑇4
in Figure 6.8 are pushed into the same primary queue at depth 𝑑. So as not to steal tasks from outside a task
group at depth 𝑑, we have to steal from queues whose depth is greater than or equal to 𝑑 within the steal range.
If a task is successfully stolen, the descendants of the task are executed by the work-ﬁrst policy and managed
in the primary queue of the thief at the same depth.
As a summary of these scheduling rules in ADWS, Figure 6.11 shows the algorithm to try to get a runnabletask when a worker has no work to do. It ﬁrst checks the local primary queues from the bottom up (line 157–
159) and then checks the local migration queues from the top down (line 160–162), so that tasks are executed72

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

156 Function GetRunnableTask()
157
158
159
160
161
162

for 𝑑 ← 𝑑max to 0 do
T ← Pop from PrimaryQueues[𝑑] of myself
if task T is popped then return T
for 𝑑 ← 0 to 𝑑max do
T ← Pop from MigrationQueues[𝑑] of myself
if task T is popped then return T// check local primary queues// check local migration queues// Begin work stealing
163
164
165
166
167
168
169
170
171
172
173
174
175

TG ← ﬁnd the dominant node in the distribution tree
if TG does not exist then return NULL
[𝑥, 𝑦) ← TG.range (i.e., current steal range)
victim ← random choice from ⌊𝑥⌋, . . . , ⌊𝑦⌋ except myself
if victim ≠ ⌊𝑥⌋ thenfor 𝑑 ← 𝑑max to TG.depth do
T ← Steal from MigrationQueues[𝑑] of victim
if task T is stolen then return T
if victim ≠ ⌊𝑦⌋ thenfor 𝑑 ← TG.depth to 𝑑max do
T ← Steal from PrimaryQueues[𝑑] of victim
if task T is stolen then return T
return NULL// steal from migration queues// steal from primary queues

Figure 6.11.: Algorithm for getting a runnable task in ADWS. This function is repeatedly called in a scheduler
loop until a task is successfully popped.
from left to right in Figure 6.8. If no local task is found, it then tries to steal work from the current steal
range. When a worker is not dominated by any task group, it does not perform work stealing (line 164), which
prevents tasks migrated by deterministic task allocation from being stolen too soon. When stealing a task,a worker ﬁrst checks the migration queues from the bottom up (line 168–170) and then checks the primary
queues from the top down (line 172–174), which is the reverse order of local pop operations2 , so that therightmost tasks are preferably stolen. Note that, in an actual implementation, we can reduce the number ofqueue checks by remembering which queues are empty. The GetRunnableTask function is repeatedly called
in a worker’s scheduler loop (spin loop) until it succeeds.

6.3.3. Issue of Shared Cache Utilization
The deterministic task mapping straightforwardly matches the task hierarchy to the cache hierarchy so thatadjacent workers that are close in the hardware topology can exploit hierarchical data locality; however, there
is still room for ADWS to further exploit hierarchical data locality on shared caches. Let us consider a casein which a task group is mapped to workers sharing a cache, and the descendants of the task group access thesame working set of data many times. Even though the descendants are executed on the same cache, if theworking set size is larger than the capacity of the shared cache, parallel execution of them can cause many
capacity misses. For a shared cache, it is often preferable to execute on a small working set at the same time(cf. cache blocking in general), so that the working set size does not exceed the cache capacity. On the other
hand, the deterministic task mapping in ADWS divides the whole computation graph vertically and assigns
2 Steal and migration operations take place on a diﬀerent side of each queue than local push/pop operations, as illustrated in Figure 6.8.

73

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

Figure 6.12.: Example of a tree of caches with 16 cores. A node denoted by 𝐶𝑖𝑙 is the level-𝑙 cache with index
𝑖, and one denoted by 𝑊 𝑘 is a worker that is pinned to the core with private cache 𝐶 𝑘2 .
tasks to workers in one go, so that the data sharing among workers becomes as minimal as possible.
For the example of decision tree construction in Section 6.2.1, suppose that the overall data (rows in
Figure 6.1) do not ﬁt into a shared cache, but the partitioned data for each subtree (rowsL and rowsR) do ﬁt
into the shared cache. In this case, parallel execution of the two tasks for constructing subtrees would cause
many more capacity misses than one-by-one execution of them. Thus, if we execute one of the two subtasks
after the other one is completed, we can promote data reuse within each task on the shared cache, which would
naturally occur in serial execution.

6.4. Multi-Level SchedulingTo address the issue of shared cache utilization in ADWS, we introduce multi-level scheduling, a genericscheduling framework for promoting data reuse on shared caches. In this section, before we introduce
multi-level ADWS in Section 6.5 by combining multi-level scheduling with ADWS, we explain the design ofmulti-level scheduling, which is orthogonal to that of ADWS. We borrow the idea of two-level scheduling [128]–[132], which is designed for speciﬁc hardware conﬁgurations and speciﬁc scheduling strategies at each cachelevel, and we generalize it to a general scheduling framework for arbitrary cache hierarchies and arbitraryscheduling policies for each cache level.
The core idea of multi-level scheduling is to limit the working set size of tasks that are simultaneously
scheduled on a shared cache so that it does not exceed the shared cache capacity. Here, we require another hintprovided by programmers; the maximum space usage of task groups must be known for performing multi-level
scheduling. In multi-level scheduling, when a task group that ﬁts into a shared cache is generated, it is assigned
to the shared cache and executed only by the workers sharing the cache. This assignment is hierarchicallyapplied to hierarchical caches with any number of cache levels. Although it may decrease the amount ofparallelism available to workers, it has the beneﬁt of increasing the shared cache utilization for memory-boundprograms in which each task group intensively reuses data among its descendants.

6.4.1. Preliminaries
To explain the design of multi-level scheduling more speciﬁcally, we ﬁrst model a cache hierarchy as a treeof caches [139]. We assume that leaf nodes are private caches (e.g., L1 caches) and that the root node is themain memory, which is of inﬁnite size. Figure 6.12 shows an example of such a tree of caches. A cache 𝐶𝑖𝑙
is identiﬁed by its cache level 𝑙 and an index 𝑖 among the level-𝑙 caches. The root node (main memory) isdenoted by 𝐶00 , given that the root cache level is 0. Note that the cache levels are numbered here in the reverse74

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
order of the prevalent naming convention for L1, L2, and L3 caches. A worker 𝑊 𝑘 is pinned to the core withprivate cache 𝐶 𝑘2 .
For multi-level scheduling, the working set sizes for task groups must be known in advance. Existing studies
have also used hints on space usage or working set sizes to improve the scheduling of nested parallel programs.
For example, Chen et al. [128]–[130] proposed an online proﬁling method to estimate the working set sizes for
iterative computations, and space-bounded schedulers [117], [125], [126] use programmer-provided hints on
the maximum space usage for each task. In this paper, we adopt the latter approach for simplicity. Figure 6.5
shows the programming style of multi-level ADWS (which will be introduced in Section 6.5), which accepts
a space usage parameter (space) for each task group3 , in addition to the work hints for each task required by
ADWS. Note that multi-level scheduling itself does not require the work hints. The working set size of a task
group TG that is speciﬁed by a programmer is denoted by Size(TG), and the capacity of a cache 𝐶 is denotedby Capacity(𝐶).

6.4.2. Design and Algorithm
In multi-level scheduling, tasks are cascaded from the top down in the tree of caches. To explain the design
and algorithm of multi-level scheduling, we ﬁrst focus on the scheduling between a level-𝑙 cache and its childcaches at level 𝑙 + 1. If the working set size of a task group is less than or equal to the cache capacity, then thetask group can be tied to the cache. Once a task group is tied to a cache 𝐶, the task group’s descendants are
all executed by the workers sharing 𝐶. At most one task group is tied to each cache simultaneously. While atask group is tied to cache 𝐶𝑖𝑙 at cache level 𝑙, the task group’s descendants are scheduled by the children of 𝐶𝑖𝑙
𝑙+1
at level 𝑙 + 1. If a child cache 𝐶 𝑙+1
𝑗 ﬁnds a task group 𝑇𝐺 for which Size(TG) ≤ Capacity(𝐶 𝑗 ), then TG is
𝑙+1
tied to 𝐶 𝑙+1
𝑗 and scheduled recursively by the children of 𝐶 𝑗 .
We can choose any strategy to schedule tasks among the children of 𝐶𝑖𝑙 . This is simpliﬁed by introducing
cache leaders and considering the scheduling among the cache leaders at each cache level. Speciﬁcally, atmost one of the workers sharing a cache acts as its leader. At the beginning of execution, leaders are electedfrom the bottom up in the cache tree; that is, the leader of a level-𝑙 cache is chosen from the leaders of its
children at level 𝑙 + 1. If a worker becomes the leader at cache level 𝑙, it stops being the leader at level 𝑙 + 1.
Note that cache leaders can change during execution and can be absent, and a worker can only be the leaderof at most one cache at a time. After the leader of the root cache 𝐶00 is chosen, it starts to run the root task
(or the main function). If a worker 𝑊 is the current leader of cache 𝐶𝑖𝑙 when it encounters a task group TG
such that Size(TG) ≤ Capacity(𝐶𝑖𝑙 ), then TG is tied to 𝐶𝑖𝑙 . At the same time, 𝑊 stops being the leader of 𝐶𝑖𝑙and becomes the leader of the next-level (𝑙 + 1) cache to which 𝑊 belongs. Then, the descendants of TG arescheduled by the leaders of the children of 𝐶𝑖𝑙 . When TG is completed, the leader of 𝐶𝑖𝑙 is elected again fromthe leaders of the children of 𝐶𝑖𝑙 .
Figure 6.13 shows the algorithm for executing a task group TG. The function ExecuteTaskGroup is awrapper for each task group’s execution (i.e., child tasks of TG are spawned and awaited at line 181 or line 184),
which is independent of the scheduling strategies for each cache level. All that must be done is to determine
whether to tie TG to the child cache. The cache for which worker 𝑊 is currently the leader is retrieved byW.cache (line 178), and we denote it as 𝐶. If TG is tied to 𝐶, then 𝑊 stops being the leader of 𝐶 and moves
on to the next cache level by becoming the leader of the child cache to which 𝑊 belongs (line 180). TG isthen scheduled among the leaders of the children of 𝐶 (line 181). After TG is completed, the worker executing
3 The previous approaches [117], [125], [126], [128]–[130] assigned the hints to tasks, but this paper assigns them to task groups

instead.

75

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

176 Function ExecuteTaskGroup(TG)
177
178
179
180
181
182
183
184

𝑊 ← Worker of interest𝐶 ← 𝑊 .cache // Cache for which 𝑊 is the current leader
if 𝐶 is not a leaf and Size(TG) ≤ Capacity(𝐶) then𝑊 .cache ← A child of 𝐶 to which 𝑊 belongs
Schedule TG among the children of 𝐶 and wait
𝑊 ′ .cache ← 𝐶
else
Schedule TG among the siblings of 𝐶 and wait
Figure 6.13.: Algorithm for execution of a task group in multi-level scheduling.

(b) ADWS at cache level 2(a) ADWS at cache level 1

Figure 6.14.: Illustration of multi-level ADWS. In each case, the upper part depicts level-1 caches, and thelower part depicts level-2 caches (corresponding to Figure 6.12). Level-1 leaves (enclosed bydotted lines) are scheduled by ADWS among the level-1 caches, and each leaf is passed to a level2 cache. Its descendants are then scheduled recursively by ADWS among the level-2 caches.
the continuation of TG (which may be diﬀerent from the previous leader 𝑊) becomes the leader of the parent
cache 𝐶 (line 182). On the other hand, if TG is not tied to 𝐶, then 𝑊 continues to schedule TG among theleaders of the siblings of 𝐶 at the current cache level, and it remains the leader of 𝐶 (line 184).
A task group tied to 𝐶𝑖𝑙 is called a level-𝑙 leaf. A level-𝑙 leaf ﬁts into 𝐶𝑖𝑙 , but its parent does not, assumingthat 𝐶𝑖𝑙 has the same capacity for all 𝑖. In overviewing the scheduling at cache level 𝑙, level-𝑙 leaves appear to
be sequential tasks that are scheduled by the sibling caches at level 𝑙. In fact, each level-𝑙 leaf is scheduledby the workers sharing cache 𝐶𝑖𝑙 in parallel, but it appears to be executed sequentially by 𝐶𝑖𝑙 , as at most one
level-𝑙 leaf is tied to a cache until it is completed. Hence, the scheduling at cache level 𝑙 can be abstracted as
the scheduling for level-𝑙 leaves among the leaders of the level-𝑙 caches. In addition, as at most one level-𝑙 leaf
is tied to a cache at a time, we have only one root task for each level of scheduling, which is the same situation
as in ordinary (single-level) schedulers for nested parallel computations. Because of these characteristics, we
can easily apply existing schedulers for nested parallelism, such as work stealing and ADWS, to each cachelevel in multi-level scheduling.

6.5. Multi-Level ADWS
By combining multi-level scheduling with ADWS, we introduce multi-level ADWS. For clarity, we refer to theoriginal design of ADWS (introduced in Section 6.3) as single-level ADWS. Multi-level ADWS is a multi-level
scheduler whose scheduling policy at each cache level is single-level ADWS. The idea of multi-level ADWS

76

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

185 Function ExecuteTaskGroup(TG)
186
187
188
189
190
191
192
193
194
195

𝑙 ← Current cache level(𝑖, 𝑗) ← Integer parts of the current distribution range (𝑖 ≤ 𝑗)∑max(𝑖, 𝑗 −1)𝑙next ← 𝑙, cap ← 𝑘=𝑖
Capacity(𝐶 𝑘𝑙 )while 𝑙next < 𝑙 max and Size(TG) ≤ cap do𝑙next ← 𝑙 next + 1∑max(𝑖, 𝑗 −1) ∑cap ← 𝑘=𝑖𝑐∈𝐷 (𝐶 𝑙 , 𝑙next ) Capacity(𝑐)𝑘if 𝑙 ≠ 𝑙next thenmax(𝑖, 𝑗 −1)
Apply single-level ADWS to ∪ 𝑘=𝑖
𝐷 (𝐶 𝑘𝑙 , 𝑙 next )else
Continue to schedule TG at current cache level 𝑙

Figure 6.15.: Algorithm for cache hierarchy ﬂattening in multi-level ADWS.
is illustrated in Figure 6.14, with a cache hierarchy that corresponds to Figure 6.12. When a level-𝑙 cacheencounters a level-𝑙 leaf while level-𝑙 leaves are being scheduled by single-level ADWS among level-𝑙 caches,
the level-𝑙 leaf is passed to the next cache level 𝑙 + 1, and single-level ADWS is applied recursively at level𝑙 + 1.
A downside of multi-level scheduling is that even for computations that are small enough to ﬁt into entire
caches, they are scheduled in a multi-level way, thus causing load imbalances. For example, if the overall
working set of the root task group ﬁts into cache 𝐶𝑖𝑙 , then only the workers sharing 𝐶𝑖𝑙 would work on it, whichleaves workers outside 𝐶𝑖𝑙 idle. Hence, we introduce a technique called cache hierarchy ﬂattening, in whichpart of the cache hierarchy is logically ﬂattened to single-level caches to enable the application of single-level
ADWS over them. The rationale for this technique is that, when an overall working set ﬁts into an entire
hierarchical cache, single-level ADWS can eﬃciently exploit data locality, because the issue of shared cacheutilization (Section 6.3.3) does not apply.
We ﬁrst explain the operation of cache hierarchy ﬂattening by using the example of a tree of caches in
Figure 6.12. For example, if the overall working set size of the root task group is less than or equal to
the sum of the capacities of level-1 caches 𝐶01 , . . . , 𝐶31 , then the whole computation should be scheduled
by single-level ADWS over all workers; otherwise, the load will not be well balanced. Accordingly, we
2 ).
ﬂatten all of the level-2 caches into single-level caches and apply single-level ADWS over them (𝐶02 , . . . , 𝐶15
Moreover, cache hierarchy ﬂattening can be applied to a sub-computation for part of the cache hierarchy.
Suppose that 𝐶21 is assigned a cross-worker task with range [2.𝑥, 4.0), which means that 𝐶21 is responsible for
distributing tasks to 𝐶31 and to itself in ADWS at level 1. When 𝐶21 encounters a task group TG such that
Size(TG) ≤ Capacity(𝐶21 ) + Capacity(𝐶31 ), the task group should be scheduled by single-level ADWS over the
2 , rather than scheduling TG only within 𝐶 1 .
children 𝐶82 , . . . , 𝐶152
Figure 6.15 shows the algorithm for cache hierarchy ﬂattening in a general case. If the distribution range of
TG is [𝑥, 𝑦), then the integer parts of this range are 𝑖 = ⌊𝑥⌋ and 𝑗 = ⌊𝑦⌋ (line 187). First, we consider the total𝑙capacity of the caches within the current distribution range, 𝐶𝑖𝑙 , . . . , 𝐶max(𝑖,(line 188)4 . If Size(TG) is𝑗 −1)
𝑙is further
less than or equal to that total capacity, then the total capacity of the children of 𝐶𝑖𝑙 , . . . , 𝐶max(𝑖,𝑗 −1)checked (line 189–191). Here, 𝐷 (𝐶, 𝑙) denotes the set of level-𝑙 caches that are descendants of cache 𝐶. This
4 We do not include 𝐶 𝑙 ( 𝑗 > 𝑖) in this range (indicated by the largest index, max(𝑖, 𝑗 − 1)) because 𝐶 𝑙 can also assign a level-𝑙 leaf𝑗𝑗
for itself, which should be prioritized to cache hierarchy ﬂattening by 𝐶𝑖𝑙 .

77

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

Table 6.1.: Experimental environment for shared-memory ADWS (Oakbridge-CX supercomputer [140]).
Processor# of Physical Cores
CPU Frequency
L1 Data Dache
L2 CacheL3 Cache
MemoryCompilerOSIntel Xeon Platinum 8280 (Cascade Lake)
56 (28 cores/socket × 2 sockets)
2.7 GHz (Turbo 4.0 GHz)
32 KB/core
1 MB/core
38.5 MB/socket
192 GBGCC 7.5.0 (-O3 -march=native)
CentOS Linux 7 (3.10.0-957.el7.x86_64)

is repeated until either the total capacity of the descendants becomes smaller than Size(TG) or the cache level
reaches the maximum level 𝑙max . Then, if the next cache level 𝑙next is updated, we ﬂatten the level-𝑙next caches𝑙and schedule TG over them by single-level ADWS (line 192–195). Otherwise, we
under 𝐶𝑖𝑙 , . . . , 𝐶max(𝑖,𝑗 −1)continue to schedule TG at the current cache level without ﬂattening the caches.
Note that, even if we apply cache hierarchy ﬂattening to other scheduling strategies that are combined withmulti-level scheduling (e.g., multi-level work stealing), the beneﬁt is limited if the corresponding single-level
scheduling is not designed for hierarchical caches. For example, as conventional random work stealing would
cause many cache misses on hierarchical caches because of its randomness, cache hierarchy ﬂattening for
multi-level work stealing would also cause many cache misses. On the other hand, multi-level ADWS canbeneﬁt from cache hierarchy ﬂattening because single-level ADWS is already cache-hierarchy-aware, to someextent, and it is therefore expected to perform well over a wide range of working set sizes.

6.6. Performance Evaluation on Shared Memory
To evaluate the performance of ADWS, we conducted experiments with seven benchmarks. In short, ourprimary ﬁndings in this section are as follows:• ADWS outperforms other existing schedulers over a wide range of working set sizes by improving cacheutilization. Multi-level ADWS reduces shared cache misses and performs better than single-level ADWS
in many cases, but it has drawbacks of increased core idle times and private cache misses (Section 6.6.3).
• ADWS is tolerant to load imbalances. Even when we do not provide any work hints to ADWS and theworkload is highly unbalanced, it performs similarly or better than random work stealing (Section 6.6.4).
• ADWS can take advantage of the NUMA local allocation policy because of its deterministic scheduling.
By allocating physical memory to proper NUMA nodes, ADWS can perform better than when the
NUMA interleave policy is adopted (Section 6.6.5).

6.6.1. Experimental Settings
For evaluation, we used a two-socket Cascade Lake machine of the Oakbridge-CX supercomputer [140] at The
University of Tokyo. The machine’s conﬁguration is summarized in Table 6.1. Hyper-threading was disabled,
while Turbo Boost and Transparent Huge Pages (THP) were enabled. The -march=native option was not set

78

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
for compiling scheduler code because the resulting assembly included SIMD instructions, which slowed down
the CPU frequency.
For evaluation, we implemented ﬁve schedulers: single-level WS (SL-WS)5 , single-level ADWS (SL-ADWS),
multi-level WS (ML-WS), multi-level ADWS (ML-ADWS), and a space-bounded scheduler (SB). All of theseschedulers were implemented on MassiveThreads [15], a lightweight user-level threading library. We ported
the SB implementation published by Simhadri et al. [133], [134] to MassiveThreads. We adopted the “SB-D”
variant with a minor modiﬁcation to avoid lock contention, because in most cases it performed better thanthe “SB” variant in their paper, which used centralized task queues. For SB parameters, we set 𝜎 = 0.5
and 𝜇 = 0.2, which was the same setting as in their evaluation. For SB only, we speciﬁed the working set
sizes for tasks rather than task groups to follow their implementation. To validate the baseline performanceof MassiveThreads, we also compared it with the widely used tasking runtime Cilk Plus [103] (the version
shipped with GCC v7.5.0), because both MassiveThreads and Cilk Plus adopt work-ﬁrst random work stealing
by default; we thus expect that SL-WS and Cilk Plus would perform similarly.
In all evaluations, as many kernel-level threads (workers) as cores were created and pinned to cores by setting
their aﬃnity. Unless explicitly noted, we set the --interleave=all option to the numactl command whenrunning the experiments, which mean that physical pages were almost evenly distributed to all NUMA nodes.
Serial execution times were measured with the --localalloc option on a ﬁxed core to avoid remote memory
accesses. For each measurement, we repeated the computation 11 times within each program execution andomitted the ﬁrst run (warm-up). We also repeated the execution ﬁve times from the outside of the program;thus, we plotted the mean (geometric or arithmetic mean depending on the context) of 50 runs in total for eachpoint.
To break down the schedulers’ execution times, we proﬁled three metrics: the busy time, idle time, andoverhead. The busy time was the time during which a worker was executing a task, i.e., performing meaningful
work in a benchmark program. The idle time was the time during which a worker was searching for readytasks because no local task existed. The overhead was the remaining time besides the busy time and idle time,
which was the overhead of the scheduler itself. Because proﬁling incurs additional overheads, we conducted
proﬁling separately from the main evaluation. As we did not modify Cilk Plus, we could not obtain a proﬁling
result for Cilk Plus.
We also proﬁled cache miss counts by using the performance monitoring units (PMUs) in the Intel processors. Instead of command-line perf, we directly called the perf event open() Linux system call from thebenchmark programs to avoid counting unnecessary events (e.g., operations at initialization) and to proﬁle each
repetition separately. For private caches, we counted L2 misses by monitoring the L2 RQSTS.MISS event [141].
For shared caches, we counted L3 misses by monitoring the LLC LOOKUP.ANY event6 in uncore Caching/Home
Agent (CHA) units [142]. All of these events count the total number of cache replacements, including datareads, writes, and prefetches.

6.6.2. Benchmarks
We used seven benchmarks to evaluate the performance of the schedulers. Each benchmark is explained in thefollowing.

5 WS denotes conventional random work stealing with the work-ﬁrst policy, which is the default scheduler of MassiveThreads.
6 With the register state of Cn MSR PMON BOX FILTER0.state=0x1.

79

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Recursive Repeated Map (RRM)

This benchmark is a simpliﬁed version of the artiﬁcial benchmark that was used in the evaluation of SB [133],[134]. It takes an array of double-precision ﬂoating-point numbers as input and logically divides the arrayrecursively into two subarrays. At each recursion before dividing an array, a map function, which just multiplies
each element by a constant and adds the product to itself, is applied to the array three times. Recursion stops
when the array size becomes smaller than 32 KB. Each map function is also recursively parallelized until thearray size becomes smaller than 128 KB. An array is divided in the ratio 1 : 𝛼 at each recursion, where 𝛼 is awork ratio parameter, which determines how much the computation graph is unbalanced.
QuicksortThis benchmark implements the well-known divide-and-conquer Quicksort algorithm, by parallelizing tworecursive calls for the partitioned arrays. The partition operation is also parallelized to increase the parallelism
through double buﬀering; thus, the total working set size is double the input array size. As it involves parallel
computations for an array before recursive parallel calls, the computation pattern is similar to that of RRM.
The shape of the computation graph depends on the input and the pivot, which is chosen as the median of theﬁrst three elements in our evaluation. The cutoﬀ sizes for both recursion and partitioning were 64 KB. Eachelement was a random double-precision ﬂoating-point number.
KDTreeThis benchmark constructs a kd-tree for randomly generated three-dimensional points (double-precision). The
algorithm is similar to Quicksort and the computation graph is also irregular. The pivot value is chosen as themedian of the ﬁrst three values along an axis (x, y, or z) that is chosen in a round-robin way at each recursion
level. The algorithm keeps creating tree nodes until the array size becomes smaller than 4 KB. The cutoﬀ sizes
for both recursion and partitioning were set to 64 KB. Compared to Quicksort, KDTree is more memory-boundbecause the recursion in KDTree stops earlier than in Quicksort, which reduces the amount of computation
per memory access.
Decision Tree

This benchmark is the same as that explained in Section 6.2.1. We used the HIGGS dataset downloaded fromthe UCI machine learning repository [143], which was published by Baldi et al. [144]. The task is binaryclassiﬁcation using 28 attributes of double-precision ﬂoating-point numbers for each row. 500,000 out of
11,000,000 rows were used for testing data, and the other rows were used for training data, which had a size of
about 2 GB. The performance evaluation accounted only for the training time. For validation, we determinedthat our implementation achieved an accuracy of 72% for the testing data, while the accuracy with random
prediction was 52%. The maximum depth for a decision tree was set to 17, which achieved the best accuracy
when using all the training data. The cutoﬀ size for recursion was 64 KB, and that for parallel loops andpartitioning was 256 KB. The computation graph can be irregular, but note again that Figure 6.3 shows awell-balanced graph for simplicity.

80

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
Matrix Multiplication (MatMul)

This benchmark performs multiplication of single-precision ﬂoating-point dense matrices (SGEMM). The
algorithm is based on cache-oblivious matrix multiplication for square matrices [67], which recursively
divides a square matrix into four submatrices. The cutoﬀ size for recursion was 64 × 64, and the computation
kernel was optimized by hand for 64 × 64 matrices with AVX-512 instructions. We added a padding of 128
bytes to each row to avoid cache conﬂict with matrices whose size was a power of two, because this padding
size showed the best performance.
Heat2D

This benchmark is a simple ﬁve-point stencil computation with double buﬀering, which is parallelized byrecursively dividing a square grid into four equally sized subgrids. Heat2D has a high degree of iterative data
locality because the same computation is repeated across iterations, whereas it does not have much hierarchicaldata locality because tasks do not share much data with others within each iteration. Thus, we cannot expect
much cache reuse when the overall working set does not ﬁt into the whole caches. The cutoﬀ size for recursion
was 64 × 64. The execution time for 50 iterations was used for performance evaluation. We added a paddingof 256 bytes to avoid cache conﬂict with matrices whose size was a power of two, because this padding size
showed the best performance.
SPH

This benchmark is a 3D dam-breaking simulation using the smoothed particle hydrodynamics (SPH) method,which calculates short-range particle interactions within an eﬀective radius. The SPH computation kernelrefers to the method by Becker and Teschner [145]. The implementation was ported from FDPS [146], which
is based on an octree [147] to partition a 3D space and eﬃciently ﬁnd the neighbors of particles. The numberof particles in an octree node was given as roughly estimated work hints for ADWS, and the space hints were
calculated based on the number of particles. The performance measurement included only the force calculation
part, which is straightforwardly parallelized by traversing the octree, whereas other parts such as tree building
were excluded. Each leaf of an octree had at most 32 particles, and the results were obtained as the execution
times for ﬁve iterations.

6.6.3. Overall Performance Results
Figure 6.16 shows the speedup on 56 cores compared with the serial execution times (except for MatMul)
for diﬀerent working set sizes. For MatMul, we plotted the FLOPS instead of the speedup; the machine’s
peak performance was 8.6 TFLOPS (the CPU frequency while executing AVX-512 instructions was 2.4 GHz).
The work ratio parameter 𝛼 of RRM was set to 1.0, which generates a well-balanced computation graph.
Figure 6.17 shows a breakdown of the execution times, and Figure 6.18 shows the cache miss counts. For eachbenchmark, the largest problem size, corresponding to the rightmost point in Figure 6.16, was used for the
performance analysis results shown in Figure 6.17 and Figure 6.18.
First, the performance of SL-WS and Cilk Plus was similar in most cases, which indicates that the baseline
performance of MassiveThreads was comparable to that of Cilk Plus. This observation is also supported
by Figure 6.17, in which the overhead of MassiveThreads is so small that only the diﬀerence in schedulingdecisions (rather than the overheads) aﬀected the overall performance.

816. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

Cilk Plus

SL-WS

ML-WS

SL-ADWS

ML-ADWS

SB

30
40

20

SpeedupSpeedup

25

15
10

30
20
10

5
0

0
10M

100M

1G

100M

Working Set Size (bytes)

1G

Working Set Size (bytes)(a) RRM (𝛼 = 1.0).

(b) Quicksort.
30

30

SpeedupSpeedup

25
20

10

20
15
10
5

0

0
100M

1G

100M

Working Set Size (bytes)(d) Decision Tree.
80

4

60

2

40

Speedup5

Speedup

TFLOPS(c) KDTree.

3

40
20

1
0
2048

4096

8192

30
20
10

0
1024

1G

Working Set Size (bytes)

0

16384

100M

N=M=K

1G

10M

Working Set Size (bytes)(e) MatMul.

100M

1G

Working Set Size (bytes)(f) Heat2D.

(g) SPH.

KDTreeDecision Tree0

MatMul0

Heat2D0

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Quicksort

0

Busy Time1
Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

RRM

0Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0

Idle Time

2

1

0.5
Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0

1

0.1

Overhead

3

2

1.5

5

0.2

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0.2

2

0.3

0.4

0.2

10

0.4

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0.6

0.4

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Execution Time (s)Figure 6.16.: Speedup (or FLOPS) on 56 cores with various working set sizes. Each vertical dashed line
represents the total size of the system’s L3 caches (38.5 MB/socket × 2 sockets = 77 MB).

SPH

Figure 6.17.: Breakdown of the execution times proﬁled by the schedulers (corresponding to the rightmost
points in Figure 6.16).

82

RRM

Quicksort

KDTree1G

Decision Tree0

MatMulHeat2D

6G
4G
2G
0
1.5G
1G
0.5G
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

2G

4G
3G
2G
1G
0
3G
2G
1G
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

8G
6G
4G
2G
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0

30G
20G
10G
0
20G
15G
10G
5G
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

200M

1G
0.5G
0
600M
400M
200M
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0
400M

1G
0.5G
0
600M
400M
200M
0

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

1G
0.5G

SerialCilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

L3
L2
Misses Misses

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

SPH

Figure 6.18.: Cache miss counts proﬁled by perf system calls (corresponding to the rightmost points in
Figure 6.16).
For small working set sizes less than the total L3 cache size, ADWS outperformed the other schedulers on
all of the benchmarks. In this range of working set sizes, ML-ADWS performed almost the same as SL-ADWS
because of the cache hierarchy ﬂattening. The performance improvement of ADWS was outstanding for
Heat2D, which showed speedup by almost 80x at most by fully exploiting iterative data locality through thedeterministic scheduling. Such super linear speedup was possible because parallel execution can use multiple
L3 caches, whereas serial execution can use only one L3 cache. The other schedulers based on randomness
for scheduling underperformed ADWS because they could not exploit iterative data locality.
For large working set sizes in the RRM, KDTree, Decision Tree, and MatMul benchmarks, the multi-level
schedulers outperformed their single-level counterparts. Notably, ML-ADWS showed improvements over
SL-ADWS of 21% on MatMul with 𝑁 = 16384 and 40% on Decision Tree with full training data. Comparedwith Cilk Plus, ML-ADWS achieved 61% performance improvement on Decision Tree with full training data.
By examining Figure 6.17, we can see that ML-ADWS greatly reduced the busy times for these benchmarks,
which indicates that the cache utilization was improved. This claim is also supported by the L3 cache misscounts shown in Figure 6.18: the multi-level schedulers incurred many fewer L3 misses than the single-level
schedulers. Moreover, the numbers of L3 misses for the multi-level schedulers were almost the same as in serial
execution, which indicates that these schedulers were nearly optimal in terms of L3 cache misses. Performance
improvements of these benchmarks were due to better exploitation of hierarchical data locality by ML-ADWS;
in contrast, cache misses in Heat2D were not reduced because Heat2D has little hierarchical data locality.
For Quicksort, however, the multi-level schedulers performed worse than the single-level schedulers. The
reason is clearly apparent in Figure 6.17: the idle times of the multi-level schedulers were much longer
than those of the single-level schedulers. Figure 6.18 shows that L3 misses were reduced with the multilevel schedulers, but the eﬀect of the idle times was larger, which resulted in worse overall performance.
Accordingly, the characteristics of the benchmarks determined whether single- or multi-level scheduling was
better. We can also see increases in the idle times of the multi-level schedulers on KDTree and Decision Tree,but the beneﬁt of improved cache utilization overwhelmed the side eﬀect of increased idle times, thus resulting
in better overall performance. The long idle times of the multi-level schedulers were due to their design, in
which only one level-𝑙 leaf at a time could be tied to a cache 𝐶𝑖𝑙 . In particular, in irregular computation, level-𝑙leaves can sometimes be so small that they expose little parallelism, which leads to underutilization of thecores sharing 𝐶𝑖𝑙 .
The SB scheduler was expected to be tolerant of increases in the idle times in irregular computations, as
discussed by Blelloch et al. [117]. However, our evaluation results showed a tradeoﬀ between idleness anddata locality in SB. The execution time breakdown (Figure 6.17) shows that SB had shorter idle times than themulti-level schedulers, but it also had longer busy times. The cache miss counts (Figure 6.18) show that SB
greatly reduced the number of L3 misses as compared with the single-level schedulers, but it incurred more83

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
L3 misses than the multi-level schedulers in most cases. This was because SB tries to tie multiple tasks to acache as long as the total working size of the tied tasks is less than or equal to the cache capacity. At the sametime, however, the granularity of tasks tied to caches becomes smaller to ﬁll up the remaining capacity of thecaches, which reduces the chance of cache reuse within each task, thus incurring more L3 misses than withmulti-level scheduling.
There was another tradeoﬀ between private cache (L2) misses and shared cache (L3) misses. The trend
was that the multi-level schedulers and SB incurred more L2 misses than the single-level schedulers, but fewer
L3 misses. As SL-WS and SL-ADWS are designed so that each worker follows the serial execution orderas much as possible, both can exploit data reuse that would occur naturally in serial execution, which makes
them private-cache-friendly. In contrast, the shared-cache-aware schedulers (ML-WS, ML-ADWS, and SB)
are likely to disturb the serial execution order for each worker so as to improve the shared cache utilization.
Nevertheless, ML-ADWS had fewer L2 misses than ML-WS, because ADWS could reduce the number ofsteals through deterministic task mapping.
Comparison of the multi-level schedulers shows that ML-ADWS consistently performed better than MLWS on RRM, KDTree, Decision Tree, and MatMul. All of these benchmarks have, at each recursion level,
consecutive parallel computations whose data locality can be exploited by deterministic scheduling. The mostobvious example is the Decision Tree benchmark (see Section 6.2.1), for which multi-level ADWS showed a
16% performance improvement with respect to multi-level WS with full training data.
Overall, ML-ADWS performed better than the other schedulers on many benchmarks over a wide range ofdata sizes, but for certain benchmarks, its performance was degraded by increased idle times (Quicksort) or
increased L2 misses (SPH). These tradeoﬀs are imposed by the design of multi-level scheduling to improveshared cache utilization. Nevertheless, we can observe that either SL- or ML-ADWS performed the best on
all of the benchmarks: when ML-ADWS did not perform well, SL-ADWS performed the best, and vice versa.
A solution to the tradeoﬀ issue would be to switch between SL- and ML-ADWS (perhaps partially by cachehierarchy ﬂattening) according to the workload characteristics, such as the arithmetic intensity and the amountof parallelism. Consideration of the workload characteristics in addition to the programmer-speciﬁed working
set sizes for application of cache hierarchy ﬂattening would be an interesting future work.

6.6.4. Sensitivity to Imprecise Work Hints
In contrast to the above experiments in which appropriate work hints are given for ADWS, in this section, westudy the performance of ADWS when work hints are imprecise or unavailable. Here, we suppose that workhints are not provided by programmers, and ADWS always guesses that the child tasks in a task group havethe same amount of work. We ﬁrst use the RRM benchmark to artiﬁcially generate unbalanced computationgraphs by changing the work ratio parameter 𝛼. Without work hints from programmers, the work ratio of thetwo recursion calls in RRM is always presumed to be 1 : 1, which is a random guess without a priori knowledge
about work. Figure 6.19 shows the performance results of RRM when changing 𝛼, which divides an array inthe ratio 1 : 𝛼 at each recursion. The y-axis represents performance improvements over SL-WS, which was
calculated by 1 − 𝑇/𝑇SL−WS , where 𝑇 is the execution time of each scheduler, and 𝑇SL−WS is that of SL-WS.
For comparison, it also shows the performance results of SL- and ML-ADWS that receive an appropriate
work ratio of 1 : 𝛼, which were plotted as the same, open markers as in Figure 6.16, whereas those without
programmer-provided work hints were plotted as ﬁlled markers.
The evaluation with the working set size of 64 MB (Figure 6.19a), which ﬁts into the system’s L3 caches,shows that ADWS outperformed the other schedulers when 𝛼 was close to 1. When the computation graph84

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

ML-ADWS (w/o work hint)
Improvement over SL-WS

Improvement over SL-WS

SL-ADWS (w/o work hint)
80%
60%
40%
20%
0%
0.1

1.0

50%
40%
30%
20%
10%
0%

10

0.1

Work Ratio of Sibling Tasks

1.0

10

Work Ratio of Sibling Tasks(a) Working set size of 64 MB.

(b) Working set size of 1024 MB.

0

Quicksort

KDTreeDecision Tree

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

SPH

0.2

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Decision Tree

0.4

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWSSB

KDTreeADWS (w/ work hint) and other schedulers
ADWS (w/o work hint)
10
0.4
3
0.3
2
+2.0% -4.9%-2.7% -3.9% 5-8.6%+4.9% -0.5% 0.2-2.4% 1
0.1
0
0
0

0.6

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Quicksort

Execution Time (s)

0

-9.2%

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

5

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

10

ADWS (w/o work hint)
ADWS (w/ work hint) and other schedulers
80
8
80
60
6
60
40
-13.0% -19.8%
40-7.6% 4-30.6% -28.7% 20-35.8%+22.1%
2
20
0
0
0
Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

15

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Execution Time (ms)Figure 6.19.: Sensitivity of imprecise work hints for various unbalanced computation graphs. It shows performance improvements of the schedulers over SL-WS on the RRM benchmark with various work
ratios (𝛼 parameter). In addition to the legends in Figure 6.16, the performance of SL- and
ML-ADWS without work hints, which guess that the work ratio is 1 : 1, is shown with the ﬁlled
markers.

SPH(a) Working set sizes close to the total L3 cache size (Quick- (b) Working set sizes that are the largest sizes in Figure 6.16
sort: 64 MB, KDTree: 64 MB, Decision Tree: 89 MB,(Quicksort: 2.0 GB, KDTree: 2.0 GB, Decision Tree:
SPH: 52 MB).
4.5 GB, SPH: 2.2 GB).

Quicksort

0.1
0+15.1% +4.3%

KDTree+19.5% -5.5%0Decision Tree

1
0.5
0

-1.3%

MatMul1
0interleave

3

2
-9.0%

2
+21.7% +22.6%

Heat2D

10

local alloc
+2.7%

-0.3%

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0

+6.0%

5

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

RRM

0.2

0.2

1.5

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWSSB+22.5%-0.8%

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWSSB

0

0.4+30.2%

2

0.3

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0.2

10

0.4

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

0.6

0.4

Cilk Plus
SL-WS
ML-WS
SL-ADWS
ML-ADWS
SB

Execution Time (s)Figure 6.20.: Evaluation of ADWS when no work hints are provided by programmers. The hatched bars
represent the execution times of ADWS without any work hints, along which the performance
improvements over ADWS that was guided by programmers’ work hints are shown (we anticipate
negative values for performance improvements because no work hints were given).

SPH

Figure 6.21.: Performance comparison between the NUMA interleave policy and the local allocation policy
with ADWS. Working set sizes correspond to the rightmost points in Figure 6.16. The performance improvements over the interleave policy are shown along with the hatched bars of thelocal allocation policy for ADWS.

85

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
was highly unbalanced (i.e., when 𝛼 was far from 1), the performance of ADWS was degraded when workhints were not provided, but it still had better performance than SL-WS. For example, even with 𝛼 ∼ 10
(i.e., when the work ratio 1 : 10 was assumed to be 1 : 1), ADWS (without work hints) performed slightly
better than SL-WS. This result indicates that ADWS is tolerant to load imbalance, even when work hints are
imprecise or unavailable. This characteristic stems from the algorithm of dynamic load balancing explainedin Section 6.3.2, in which ADWS localizes work stealing as much as possible and automatically regresses
to conventional random work stealing when load imbalance is large. The evaluation with the large working
set size of 1024 MB (Figure 6.19b) shows similar results, in which ML-ADWS (w/o work hint) performed
similarly to ML-WS when the computation graph is highly unbalanced, which means that ADWS at each cachelevel shortly regressed to conventional random work stealing when 𝛼 is far from 1.
Next, we study the performance of other benchmarks: Quicksort, KDTree, Decision Tree, and SPH. We
exclude Matmul and Heat2D because the random guess for work ratios is always correct for these benchmarks
as their computation pattern is regular. Figure 6.20 highlights the changes of performance of ADWS whenwork hints are not provided by programmers. In most cases, performance improvements of ADWS (withoutwork hints) over ADWS (with work hints) were negative, which means that ADWS cannot fully exploit data
locality if precise work hints are not provided. Nevertheless, the performance of ADWS (even without workhints) was comparable to that of other schedulers such as SL-WS. These results are consistent with those ofRRM (Figure 6.19) in that the degree of performance degradation was larger for working set sizes close to thetotal L3 cache size (Figure 6.20a; up to 35.8% in Decision Tree) than for larger working set sizes (Figure 6.20b;less than 10% performance drop). Note that the performance of ML-ADWS in Decision Tree was unstable in
Figure 6.20a because the working set size was slightly larger than the total L3 cache size (89 MB > 77 MB)
and thus the number of L3-leaf tasks was too small.
These results suggest that we can use ADWS as the default scheduler for nested parallel computations even
without work hints provided by programmers. When a well-balanced computation graph is given, there isa chance to improve cache utilization by ADWS. On the other hand, when the computation graph is highly
unbalanced, it can perform dynamic load balancing as well as conventional random work stealing. While theprevious section showed that multi-level schedulers have several drawbacks, ADWS itself is robust to load
imbalance. In addition, if programmers can provide appropriate work hints, the performance can be further
improved.

6.6.5. NUMA Memory Policy for Deterministic SchedulingTo this point, we used numactl --interleave=all command, which distributes physical pages to all NUMA
nodes as evenly as possible. However, by taking advantage of the deterministic scheduling in ADWS, we
can optimize the memory mapping so that local memory accesses are increased. In this section, we study
the performance of ADWS when the NUMA memory placement is properly arranged for ADWS. To allocate
memory to NUMA nodes where the main computation will take place, we parallelized the initialization of theworking set (with a memory access pattern that resembles the main computation) in each benchmark, so thatphysical pages are mapped to proper NUMA nodes by the ﬁrst-touch policy7 .
Figure 6.21 shows performance comparison between the NUMA interleave policy and the local allocation
policy for ADWS. The working set sizes are the largest sizes in Figure 6.16. For schedulers other than ADWS,
we do not show the performance with the local allocation policy because their scheduling is not deterministic.
7 In our experiments, to prevent physical pages from being dynamically migrated by automatic NUMA balancing in Linux, we usedmbind() syscall to ﬁx physical pages to speciﬁc NUMA nodes.

866. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)

Depth
0
1
2
3
4
5
3

2

1

0

Figure 6.22.: Distributed-memory representation for a distribution tree in ADWS. This situation corresponds
to that of Figure 6.10. The red-ﬁlled nodes represent active nodes, and the nodes pointed by the
thick arrows are the dominant nodes for each worker.
Overall, SL-ADWS beneﬁts from the local allocation policy more than ML-ADWS, because SL-ADWS causes
more accesses to the main memory because of many L3 misses. Nevertheless, the performance of ML-ADWS
is improved by about 20% in RRM and Heat2D, which are highly memory-bound and regular computations.
For MatMul, which computes 𝐶 = 𝐴𝐵, we could not see performance improvements because the memory
access pattern for matrix 𝐴 and 𝐵 is not regular.

6.7. Distributed-Memory Implementation in Itoyori
Thus far, we have introduced ADWS and validated its eﬀectiveness on shared-memory computers by implementing it in MassiveThreads. In this section, we explain our distributed-memory implementation of
single-level ADWS in Itoyori. The ADWS implementation requires both child stealing and continuation stealing, as ADWS switches between the child-ﬁrst and parent-ﬁrst policies. Nonetheless, as both strategies havealready been supported in Chapter 4, it was straightforward to implement ADWS to Itoyori. Data structures
such as distribution ranges and work stealing queues are also easy to implement with Itoyori’s multithreading
layer, except for a distribution tree.
A distribution tree (Figure 6.10) is a global data structure for dynamically managing the steal range foreach worker. When a worker attempts a steal, it traverses the distribution tree from a leaf node to the root
node to determine the dominant node, which has the steal range. This operation is intended to be performed
every time the worker attempts a steal, because other workers could have activated the nodes close to the root.
However, we found that naively checking the status of tree nodes with each steal attempt resulted in congestionin communication. This is because the nodes close to the root can be frequently accessed by many workers.
Figure 6.22 shows a distributed-memory representation for the distribution tree, corresponding to thesituation in Figure 6.10. In this representation, the state for a distribution tree node with a distribution range[𝑥, 𝑦) at depth 𝑑 is located at the cell 𝐶 ⌊𝑑𝑥 ⌋ , the 𝑑-th cell in worker ⌊𝑥⌋. A node state includes the reference toits parent node, its distribution range, and the active ﬂag. When a node becomes active, the active ﬂag at thecorresponding cell is set by an RMA call by some worker. The active ﬂag is eventually read by other workers
during tree traversal. As each node has a reference to its parent node, each worker can traverse the tree froma leaf node to the root node by repeatedly issuing RMA calls to remote cells. However, this implementation87

6. Locality-Aware Task Scheduling with Almost Deterministic Work Stealing (ADWS)
leads to congestion in communication because nodes that are close to the root can be frequently accessed bymany workers.
We address this issue by distributing an active ﬂag to multiple cells in diﬀerent workers, so as to avoid
congestion in speciﬁc workers. In our implementation, the state for a node with a distribution range [𝑥, 𝑦) at
depth 𝑑 at the primary cell 𝐶 ⌊𝑑𝑥 ⌋ is duplicated in secondary cells 𝐶 𝑑𝑗 for all worker 𝑗 such that ⌊𝑥⌋ < 𝑗 < ⌊𝑦⌋.
When a node becomes active, the active ﬂag at the primary cell is activated. When worker 𝑗 checks the statusof a node with a distribution range [𝑥, 𝑦) at depth 𝑑, it randomly chooses a cell 𝐶 𝑘𝑑 from ⌊𝑥⌋ ≤ 𝑘 < ⌊𝑦⌋ andchecks its active ﬂag. If the active ﬂag of 𝐶 𝑘𝑑 is already set, worker 𝑗 sets the active ﬂag of its local cell 𝐶 𝑑𝑗 .
By following this procedure, once the active ﬂag is set at the primary cell, its state is gradually propagated tosecondary cells by randomized checking.
Finally, we explain how to duplicate node states to secondary cells. One possible approach is to broadcast
the node state to all secondary cells when a new node is created, but this puts a signiﬁcant burden on speciﬁcworkers, particularly for those who create nodes close to the root. Instead, we consider propagating duplicatedstates along with the cross-worker tasks. As it is clear from Figure 6.22, if worker 𝑖 receives a cross-worker
task at depth 𝐷 from worker 𝑗, it is suﬃcient to create duplications for its local cells {𝐶𝑖𝑑 | 0 ≤ 𝑑 < 𝐷}.
Supposing that worker 𝑗 already has the original or duplicated states at cells {𝐶 𝑑𝑗 | 0 ≤ 𝑑 < 𝐷}, worker 𝑖copies node states from remote cells {𝐶 𝑑𝑗 | 0 ≤ 𝑑 < 𝐷} to local cells {𝐶𝑖𝑑 | 0 ≤ 𝑑 < 𝐷}. As these memory
regions are contiguous, this operation can be completed with one RMA GET call.

6.8. Summary
In this chapter, we introduced two variants of ADWS to improve the cache utilization of fork-join programs:
single-level and multi-level ADWS. Single-level ADWS is based on a deterministic, cache-hierarchy-aware
schedule with a little scheduling variety to allow for dynamic load balancing. Multi-level ADWS is built
on top of single-level ADWS by combining it with multi-level scheduling, which is a general schedulingframework for improving shared cache utilization. Unlike existing schedulers, ADWS is designed to exploit
both hierarchical and iterative data locality on deep memory hierarchies, while requiring some additional
hints from programmers. Our empirical performance analysis on shared memory demonstrated that ADWS
improved the cache utilization on many fork-join programs over a wide range of working set sizes. Aninteresting lesson that we learned from the experiments is that there are performance tradeoﬀs between singleand multi-level ADWS: multi-level ADWS incurs more private cache misses and longer core idle times, but it
can greatly reduce shared cache misses.
In addition, we implemented single-level ADWS in Itoyori by resolving the congestion issue in the distribution tree data structure used for dynamic load balancing. In the next chapter, we will conduct performance
evaluation of ADWS on large-scale distributed-memory computers.

88

7. Application Studies and Performance Evaluation
The primary goal of our performance evaluation is to demonstrate that the fork-join model can successfully
scale to distributed memory with the help of software caching and ADWS. To this end, we ran several
fork-join applications with Itoyori: Cilksort (Section 7.2), Heat (Section 7.3), UTS-Mem (Section 7.4),
ExaFMM (Section 7.5), ANNS graph construction (Section 7.6), and PageRank (Section 7.7), which all
involve a reasonable amount of global memory access (unlike the evaluation of distributed multithreading in
Section 4.5). These applications are written in a shared-memory-like, global fork-join model using Itoyori
APIs. No explicit load balancing is performed in their code.

7.1. Experimental Settings
In our evaluation, we used two supercomputers: SQUID supercomputer (CPU nodes) at Osaka University(SQUID-C) [148] and Wisteria/BDEC-01 supercomputer (Odyssey subsystem) at the University of Tokyo(Wisteria-O) [100]. Table 7.1 summarizes the conﬁguration of SQUID-C and Wisteria-O. SQUID-C
is a computing cluster with Intel Xeon CPUs and InﬁniBand. Wisteria-O is conﬁgured similarly to thesupercomputer Fugaku [101], which consists of Fujitsu A64FX CPUs and Tofu interconnect D. A computing
system based on Tofu interconnect D is conﬁgured in units of 12 nodes. Therefore, on Wisteria-O, we
allocated nodes in multiples of 12 as much as possible for better communication performance. In addition, tominimize the average communication cost, we speciﬁed a topology with a shape close to a cube. Therefore, werequested the system job scheduler to allocate 1, 2, 6 (2 × 3), 12 (2 × 3 × 2), 36 (3 × 4 × 3), 144 (6 × 6 × 4), and576 (8 × 9 × 8) nodes as a torus topology. Unless explicitly noted, we use all cores in each node by spawning
as many MPI processes as the number of cores in each node.
For inter-node communication on SQUID-C, we used Open MPI 5.0.0 built with UCX 1.15.0 as the MPI3 RMA layer by setting the osc ucx acc single intrinsic option (see Section 4.4.1). On Wisteria-O,
we used a vendor-provided MPI implementation (Fujitsu MPI). Nevertheless, for performance reasons, webypassed MPI and directly used the uTofu library (low-level APIs for Tofu interconnect D) for Itoyori’s PGAS

Table 7.1.: Experimental environment for Itoyori application studies.

Processor
CPU Frequency# of Physical Cores
MemoryInterconnect
MPI
CompilerOS

SQUID-C [148]

Wisteria-O [100]Intel Xeon Platinum 8368 (Ice Lake)
2.4 GHz (Turbo 3.4 GHz)76 (38 cores/socket × 2 sockets)
256 GB
InﬁniBand HDR (200 Gbps)
Open MPI 5.0.0
GCC 12.2.0 (-O3)
Rocky Linux 8.6

Fujitsu A64FX (ARMv8.2-A + SVE)
2.2 GHz48
32 GB
Tofu Interconnect D
Fujitsu MPI 4.0.1
Fujitsu Compiler 4.10.0 (-O3 -Nclang)
Red Hat Enterprise Linux 8.3

89

7. Application Studies and Performance Evaluation
communication. In contrast, on SQUID-C, we used MPI for all communication.
Itoyori’s parameters were conﬁgured as follows. The memory block size was set as 64 KB (the minimum
page size in Wisteria-O) on both SQUID-C and Wisteria-O. The software cache size was set as 128 MB
per process and the sub-block size was set as 4 KB unless explicitly noted. If random work stealing is
performed, we allocated collective memory in the block-cyclic distribution policy. This is to distribute thecommunication across diﬀerent nodes as much as possible, due to the unpredictability of task placement. In
contrast, for ADWS, we used the block distribution policy for collective memory allocation, so that its task
mapping matches the memory mapping as much as possible. This situation is essentially the same as the
NUMA memory allocation discussed in Section 2.1.2 and Section 6.6.5.
As a baseline for the naive integration of the PGAS and fork-join models, we implemented GET/PUT
APIs without caching in Itoyori. We consider this GET/PUT implementation as a reasonable baseline for
performance comparison, because these GET/PUT APIs are merely a thin wrapper for MPI calls (MPI Get()
and MPI Put()), and MPI-3 RMA is also considered a PGAS library.
For performance evaluation, we repeated executions for 10 times after one warm-up run and plotted theirmean and the 95% conﬁdence interval as error bars, which were suﬃciently small in most cases. Serial
execution times for calculating speedups were measured by eliding all Itoyori runtime calls (e.g., fork/join,
checkout/checkin) from the program. We also implemented a performance proﬁler in Itoyori to analyse thebreakdown of execution times. As the proﬁler introduces nonnegligible overheads particularly for ﬁne-grained
parallelism and frequent checkout/checkin calls, we separated the proﬁling run from the primary evaluation.

7.2. Cilksort
Cilksort is a recursive parallel merge sort algorithm originally included in the Cilk program example [9], whichis essentially the same as the algorithm previously shown in Figure 3.6. Cilksort is a more advanced algorithm,
in which the merge operation is also recursively parallelized. When merging two sorted arrays, the task canbe divided into two independent subtasks if we can ﬁnd the appropriate division point for each array. Supposethat we are merging two sorted arrays 𝐴 and 𝐵, and for a given value 𝑝, 𝐴 can be divided into 𝐴1 (𝑎 < 𝑝 for
all 𝑎 ∈ 𝐴) and 𝐴2 (𝑎 ≥ 𝑝 for all 𝑎 ∈ 𝐴2 ), and 𝐵 can be divided into 𝐵1 (𝑏 < 𝑝 for all 𝑏 ∈ 𝐵) and 𝐵2 (𝑏 ≥ 𝑝
for all 𝑏 ∈ 𝐵2 ). At this time, the merging task can be divided into merging for 𝐴1 and 𝐵1 and merging for 𝐴2
and 𝐵2 , which can be performed in parallel. To ﬁnd such appropriate division points, Cilksort performs binary
search for the sorted arrays. As it uses double buﬀering for parallelized merging, it requires twice the memoryof the input array. As well as the merge sort example in Figure 3.6, this divide-and-conquer algorithm can beconcisely written in Itoyori. The full program example of Cilksort is explained in Section A.1.

Caching Policies and Single-Node PerformanceFirst, we evaluate diﬀerent caching policies by varying the task cutoﬀ count (the cutoff value). Hereafter,
we consider the time to sort an array of 4-byte integers that were generated uniformly at random. Figure 7.1
compares four diﬀerent caching policies in Itoyori: No-Cache (GET/PUT), Write-Through, Write-Back, andWrite-Back (Lazy). No-Cache (GET/PUT) is the GET/PUT implementation without caching, executed byreplacing the checkout/checkin calls with the GET/PUT calls by allocating user buﬀers for them. Write-Throughand Write-Back represent the write-through and write-back cache discussed in Section 5.3.4. Write-Back (Lazy)means the lazy write-back policy for release fences (Section 5.4.2). The results on multiple nodes (Figure 7.1aand Figure 7.1b) clearly show that the more we delayed the write-back operations, the better performance90

No-Cache (GET/PUT)
Write-ThroughWrite-Back
Write-Back (Lazy)

10

Execution time (s)Execution time (s)7. Application Studies and Performance Evaluation

1

No-Cache (GET/PUT)
Write-ThroughWrite-Back
Write-Back (Lazy)

10

1
100

1000

10k

Cutoff count (# of elements)

100k

100

(a) SQUID-C, 8 nodes, 608 cores (log-log graph).
No-Cache (GET/PUT)
Write-ThroughWrite-Back
Write-Back (Lazy)
MassiveThreads

4
3.5

10k

100k

(b) Wisteria-O, 12 nodes, 576 cores (log-log graph).
No-Cache (GET/PUT)
Write-ThroughWrite-Back
Write-Back (Lazy)
MassiveThreads

25

Execution time (s)Execution time (s)

4.5

1000

Cutoff count (# of elements)

3
2.5
2

20
15
10
5

1.5
100

1000

10k

Cutoff count (# of elements)

100k

100

1000

10k

Cutoff count (# of elements)(c) SQUID-C, 1 node, 76 cores (semi-log graph).

100k(d) Wisteria-O, 1 node, 48 cores (semi-log graph).

Figure 7.1.: Execution time of Cilksort (1G elements) with various cutoﬀ counts and caching policies. “NoCache (GET/PUT)” represents conventional GET/PUT APIs (without caching) implemented in
Itoyori. “Write-Through”, “Write-Back”, and “Write-Back (Lazy)” are caching policies of Itoyori
with checkout/checkin APIs. “MassiveThreads” is an existing user-level threading library forshared memory [15]. All of them were scheduled by random work stealing (work-ﬁrst policy).

91

7. Application Studies and Performance Evaluation

10

10G elements

Execution time (s)Execution time (s)10G elements

1
1G elements

10
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin

1

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

1G elements

HykSort (MPI+OpenMP)

0.1

0.1
1

10

# of nodes (× 76 cores/node)(a) SQUID-C.

1

10

100

# of nodes (× 48 cores/node)(b) Wisteria-O.

Figure 7.2.: Execution time of Cilksort with strong scaling. The straight dashed lines represent the ideal
speedup (calculated based on the serial std::sort() execution time) for 1G and 10G elements
respectively. Exceptionally, for the execution of 10G elements on Wisteria-O, the ideal speedup
is calculated based on the best execution time on 6 nodes.
we got on multiple nodes. In particular, when the cutoﬀ count was as low as 64, Write-Back (Lazy) was
1.46×, 1.91×, and 14.7× faster than Write-Back, Write-Through, and No-Cache (GET/PUT), respectively, on
Wisteria-O. This demonstrates that Write-Back (Lazy), which faithfully adheres to the work-ﬁrst principle, isthe most robust to ﬁne-grained parallelism.
In addition, Figure 7.1c and Figure 7.1d show the single-node performance. Since caching is not involvedin a single-node execution, there is no diﬀerence between Write-Through, Write-Back, and Write-Back (Lazy).
Nevertheless, No-Cache (GET/PUT) incurs overheads even on a single node, particularly on Wisteria-O(up to 1.6× slower). This is because of the additional memory copy between the user buﬀer and the home
blocks, as discussed in Section 5.2. For reference, we executed a genuine shared-memory program of Cilksort
with MassiveThreads [15], a user-level threading library that supports child-ﬁrst random work stealing. The
performance of Itoyori is close to that of MassiveThreads, although Itoyori is slightly slower on SQUID-C(by up to 1.2×). Despite this small overhead to support distributed-memory execution, Itoyori is suﬃciently
eﬃcient even compared with an existing shared-memory fork-join runtime system.

Strong Scaling
Then, we conducted a strong scaling study for two array sizes (1G and 10G elements) with the best-performing
cutoﬀ count of 16K on Wisteria-O (12 nodes). Hereafter, we show only Write-Back (Lazy) as the cacheenabled version because the performance diﬀerence was subtle with suﬃciently large cutoﬀ counts. Figure 7.2
shows the execution time and Figure 7.3 shows the parallel eﬃciency with strong scaling. They compare
diﬀerent scheduling algorithms: ADWS and random work stealing (“Random WS”), and diﬀerent PGAS
implementations: GET/PUT APIs (without caching) and checkout/checkin APIs (with caching). The straightdashed lines represent the ideal linear speedup calculated based on the serial std::sort() execution timefor each array size. The parallel eﬃciency was also calculated based on the serial std::sort() execution
time. Notably, Itoyori (ADWS, Checkout/Checkin) achieved 47% parallel eﬃciency on 32 nodes (2,432 cores)on SQUID-C compared to std::sort() for 10G-element arrays. This is calculated to 1, 133× speedup on
2,432 cores. When compared to the serial execution time of Cilksort, it was 54% parallel eﬃciency and a
1, 302× speedup on 2,432 cores. Comparing diﬀerent execution policies, ADWS was 2.2× faster than random92

7. Application Studies and Performance Evaluation
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin

HykSort (MPI+OpenMP)

80
60
40
20
0

100
10G elements

80
60
40
20
0

1

10

# of nodes (× 76 cores/node)(a) SQUID-C (1G elements).

Parallel Efficiency (%)

100
1G elements

Parallel Efficiency (%)

Parallel Efficiency (%)

100

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

1G elements
80
60
40
20
0

1

10

1

10

100

# of nodes (× 76 cores/node)# of nodes (× 48 cores/node)(b) SQUID-C (10G elements).

(c) Wisteria-O (1G elements).

Figure 7.3.: Parallel eﬃciency of Cilksort with strong scaling. Parallel eﬃciency for the executions in Figure 7.2
is calculated based on the serial std::sort() execution time. The execution of 10G elements on
Wisteria-O is omitted because serial execution time is not available.
work stealing on 32 nodes on SQUID-C. Software caching improved performance by 36% on 36 nodes on
Wisteria-O with 10G elements, but not with 1G elements. This is because 10G elements oﬀered abundantparallelism, which enhanced cache reuse among multiple tasks within each process. Note that sorting for 10G
elements was not feasible on one and two nodes on Wisteria-O due to the out-of-memory error. At the sametime, this demonstrates that the multi-node execution with Itoyori can handle working sets larger than the
single-node memory (32 GB), as it requires at least 10G × 4 bytes (integer) × 2 (double buﬀering) = 80 GBof memory.
To investigate the cause of the performance diﬀerences between diﬀerent execution policies, we conducted
performance proﬁling in Itoyori. Figure 7.4 shows the execution time breakdown, which is normalized tothe total execution time. The legend Sched Overheads corresponds to the scheduling overheads including
fork/join calls, but excluding the Sched Loop time. The Sched Loop time is the accumulated time during
which workers are in the scheduler loop, searching for available tasks because they have no local tasks. If
the ratio of the Sched Loop time is large, it suggests that parallelism is not enough. The legends Get, Put,
Checkout, Checkin, Release, and Acquire correspond to Itoyori PGAS APIs: GET/PUT APIs, checkout/checkinAPIs, and release/acquire fences. Note that GET/PUT APIs may be used even when checkout/checkin APIs
are available; for example, Cilksort uses the GET interface in the binary search function (Figure A.2 in
Section A.1). This is because GET/PUT APIs are sometimes more convenient and eﬃcient, particular forsmall values, as checkout/checkin APIs always involve two API calls. Therefore, in Itoyori (with caching),
Get and Checkout include communication time for reading remote data. On the other hand, PUT and Checkinusually do not involve communication because the write-back operations for dirty data are mostly delayed
until inter-process synchronization (if cache is enabled). Instead, Release or Acquire includes communication
for write operations. The Release time includes both eager and lazy write-back operations. The Acquiretime is mostly the wait time for the lazy release operation, but it also includes some write-back operations
because Itoyori ﬂushes all dirty data to their home before cache invalidation. Finally, the legend User Program
corresponds to the computation time in user threads, excluding all PGAS APIs. If the ratio of User Programis large, it means that workers spend their time in meaningful work.
The single-node executions in Figure 7.4 show high User Program ratios (more than 0.8), because nointer-node communication is involved. Nevertheless, executions with GET/PUT (no-cache) show slightly

93Time ratio

7. Application Studies and Performance Evaluation

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

32

0

1

2

4

8

16

32

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

32

1

0

ADWS, Checkout/Checkin

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, 1G elements.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

32

0

1

2

4

8

16

32

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

32

1

0

ADWS, Checkout/Checkin

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, 10G elements.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36 144

0

1

2

6

12

36 1440

ADWS, GET/PUT (no-cache)

1

2

6

12

36 1441

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

36 144# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, 1G elements.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

6

12

36

144

0

6

12

36

144

0

ADWS, GET/PUT (no-cache)

6

12

36

144

1

0

ADWS, Checkout/Checkin6# of nodes (× 48 cores/node)(d) Wisteria-O, 10G elements.

Figure 7.4.: Proﬁling of Cilksort with strong scaling.

94

12

36

144

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program8

Execution time (s)Execution time (s)7. Application Studies and Performance Evaluation

6
4

40
30

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin

20

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

10

2

HykSort (MPI+OpenMP)

1

10

1

# of nodes (× 76 cores/node)

10

100

# of nodes (× 48 cores/node)(a) SQUID-C.

(b) Wisteria-O.

Figure 7.5.: Execution time of Cilksort with weak scaling (1G elements per node).
100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin40

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin20

HykSort (MPI+OpenMP)

0
1

10

# of nodes (× 76 cores/node)(a) SQUID-C.

1

10

100

# of nodes (× 48 cores/node)(b) Wisteria-O.

Figure 7.6.: Parallel eﬃciency of Cilksort with weak scaling (1G elements per node). Parallel eﬃciency forthe executions in Figure 7.5 is calculated based on the best one-node execution time among theexecution policies.
larger GET and PUT ratios, which suggests that GET/PUT APIs incurred additional memory copy overheads
between the user buﬀers and the home blocks. On multiple nodes, the time ratios of PGAS APIs increase
because of inter-node communication. Clearly, ADWS incurs less communication because of improved data
locality. The main source of overheads in ADWS, Checkout/Checkin is the GET time, meaning that binary
search spent a considerable amount of time. Because the binary search operation can be performed for large
arrays, due to its low locality access, it is diﬃcult for the cache to be eﬀective. In addition, as the number ofnodes increases, the Sched Loop ratio also increases particularly for 1G elements. This situation suggests thatparallelism is not suﬃcient on these number of nodes.

Weak Scaling
In strong scaling, executions on large node counts suﬀered from insuﬃcient parallelism. Next, we conduct
weak scaling experiments by increasing the input array size proportionally to the number of nodes. In thisevaluation, we set the input array size as 1G elements per node. Although we usually expect the execution
time to be constant in weak scaling experiments, this is unlikely to occur in Cilksort because the merge sortalgorithm requires 𝑂 (𝑛 log 𝑛) work.
Figure 7.5 shows execution time and Figure 7.7 shows parallel eﬃciency in weak scaling. The parallel95

7. Application Studies and Performance Evaluation
Random WS, GET/PUT (no-cache) Random WS, Checkout/CheckinExecution time (s)8ADWS, GET/PUT (no-cache)

6

2.5

6

2

2

4

1.5

4

0

1

1

2

2

ADWS, Checkout/Checkin
3

3

0.5
1

2

4

8

16

32

0

1

2

4

8

16

0

32

1

2

4

8

16

32

0

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)(a) SQUID-C
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin

ADWS, GET/PUT (no-cache)

ADWS, Checkout/CheckinExecution time (s)

25
40
20
30
20

10

5

5

10

10
0

10

15

5
1

2

6

12

36 144

0

1

2

6

12

0

36 144

1

2

6

12

36 144

0

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

36 144# of nodes (× 48 cores/node)(b) Wisteria-O

Figure 7.7.: Proﬁling of Cilksort with weak scaling (1G elements per node).
eﬃciency was calculated based on the best execution time among the single-node executions. The results showthat ADWS with checkout/checkin APIs performed the best. Even on 32 nodes on SQUID-C, it kept about
60% parallel eﬃciency. Figure 7.7 shows the proﬁling result in weak scaling. In contrast to the strong scaling
proﬁling (Figure 7.4), it shows execution times as y-axis. The User Program time slightly increases as thenumber of nodes increases (considered to be the 𝑂 (𝑛 log 𝑛) eﬀect), but more interestingly, the communication
time (GET and Checkout) substantially increases. This is because the merge and binary search operations are
not very cache-friendly for large arrays, and the input arrays become huge in weak scaling.

Comparison with HykSort (MPI+OpenMP)
We also compared the performance of Cilksort with that of HykSort [149], a highly optimized distributedmemory sorting algorithm. We used the author-provided C++ implementation of HykSort (included in “Utah
Sorting Library (usort)” [150]). In HykSort, each MPI process ﬁrst sorts a subarray that has been assigned toitself, and then it exchanges parts of the sorted subarray with other processes to get the overall sorted array(still distributed to multiple processes). Operations in each MPI process are further parallelized by parallel for
loops of OpenMP. Therefore, HykSort employs two levels of parallelization (MPI+OpenMP). In experiments,
we spawned one MPI process per node and launched as many OpenMP threads as the number of cores for each
node.
Strong scaling plots (Figure 7.2 and Figure 7.3) and weak scaling plots (Figure 7.5 and Figure 7.6) also plot theperformance of HykSort. Overall, Itoyori (ADWS, Checkout/Checkin) performed competitively to HykSort,
and in many cases, Itoyori outperformed HykSort. However, what is more important than which is faster, isthat even with a very concise algorithm description like Cilksort, Itoyori can demonstrate performance andscalability on par with manually load-balanced and communication-optimized implementations like HykSort.

96

7. Application Studies and Performance Evaluation
The conciseness of Cilksort compared with HykSort can be demonstrated by counting lines of code (LOC).
We extracted the code needed to perform parallel sorting, excluding code for input array initialization, result
validation, proﬁling, etc. In addition, lines that are not directly related to the operation of the parallel
algorithm itself were omitted. For example, we ignored lines for comments, blank lines, assertions, include
directives, namespace deﬁnitions, forward deﬁnitions of functions, and so on. Finally, we applied a code
formatter (clang-format 14.0.6) to both programs. Note that both Cilksort and HykSort use std::sort() andstd::merge() for sequential sorting and merging; they do not have manually optimized kernel implementations
for them. As a result, HykSort was 392 LOC, while Cilksort was only 69 LOC. The main source of complexityof HykSort was the complicated communication procedure and two levels of parallelization for MPI and
OpenMP. In contrast, Cilksort with Itoyori is written with a single, global-view fork-join model, without any
explicit load-balancing and communication code.

7.3. HeatThe Heat benchmark is also included in the Cilk program examples [9]. This program is a simple ﬁve-pointstencil computation for two-dimensional heat transfer simulation. The input matrix size is represented as
𝑁 𝑥 × 𝑁 𝑦 , where 𝑁 𝑥 is the number of elements along with the x-axis, and 𝑁 𝑦 is that along with the y-axis. Eachelement is a double-precision ﬂoating-point number, and the values are arranged along the y-axis in memory.
The stencil computation is recursively parallelized by dividing the x-axis domain into two equal-sized domains
until the x domain reaches a cutoﬀ count. In our experiments, we simply set the cutoﬀ count as one.
The Heat benchmark can be easily ported to Itoyori as well as Cilksort, while keeping the original program
structure and algorithm. The Heat benchmar is highly memory-bound because it has little work per memory
load/store operation. In addition, its computation and memory access patterns are regular and easy to predict.
Therefore, the Heat benchmark is expected to signiﬁcantly beneﬁt from ADWS.
For performance comparison, we also provided an MPI+OpenMP version of the Heat benchmark, which
is a very simple, textbook-level implementation. In this implementation, each MPI process exchanges its
halo regions with its two neighbors at each iteration, and then it computes on the local memory. The stencil
computation in each MPI process is simply parallelized by the OpenMP parallel for loop (the static policy)
along with the 𝑥 axis.

Strong Scaling
Figure 7.8 and Figure 7.9 show the execution time and parallel eﬃciency for (𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K). We
plotted the execution time for one iteration in the graphs. The straight dashed lines represent the ideal linear
speedup, which is calculated based on the serial execution time. As the Heat benchmark is highly memorybound, the single-node executions already show low parallel eﬃciency compared to the serial execution time.
For example, MPI+OpenMP only shows a 17.3× speedup on SQUID-C (76 cores) and a 23.4× speedup on
Wisteria-O (48 cores). This is why we calculated parallel eﬃciency in Figure 7.9 based on the best single-node
execution time (MPI+OpenMP).
The single-node performance of Itoyori is considerably diﬀerent across execution policies. First, GET/PUT
APIs are much slower than checkout/checkin APIs (3.4× slower on Wisteria-O) because of memory copy
overheads, which signiﬁcantly aﬀect memory-bound programs like Heat. This is also supported by the proﬁling
result in Figure 7.10, showing large GET/PUT time ratios in single-node executions. In addition, even withcheckout/checkin APIs, ADWS outperforms random work stealing (by 1.6× on SQUID-C). This is because of97

7. Application Studies and Performance Evaluation

Execution time (s)Execution time (s)

1
0.1

0.01

0.1
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin

0.01

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

0.001

MPI+OpenMP

0.001
1

10

1

# of nodes (× 76 cores/node)

10

100

1000

# of nodes (× 48 cores/node)(a) SQUID-C.

(b) Wisteria-O.

Figure 7.8.: Execution time of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)). The straightdashed lines represent the ideal speedup (calculated based on the serial execution time).

100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin40

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin20

MPI+OpenMP

0
1

10

# of nodes (× 76 cores/node)(a) SQUID-C.

1

10

100

# of nodes (× 48 cores/node)(b) Wisteria-O.

Figure 7.9.: Execution time of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)). Paralleleﬃciency of the executions in Figure 7.8 is calculated based on the best one-node execution time
among the execution policies.

98Time ratio

7. Application Studies and Performance Evaluation
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

32

0

1

2

4

8

16

0

32

ADWS, GET/PUT (no-cache)

1

2

4

8

16

32

1

0

ADWS, Checkout/Checkin

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12 36 144 576

0

1

2

6

12 36 144 576

0ADWS, GET/PUT (no-cache)

1

2

6

12 36 144 576

1

0ADWS, Checkout/Checkin

1

2

6

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

12 36 144 576# of nodes (× 48 cores/node)(b) Wisteria-O

Figure 7.10.: Proﬁling of the Heat benchmark with strong scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K, 32K)).
improved data locality in NUMA architectures (see also Section 6.6). These results demonstrate that ADWS
and checkout/checkin APIs are designed not only for scalability on multiple nodes but also to eﬃciently utilize
shared-memory resources.
On multiple nodes, the performance of random work stealing signiﬁcantly drops because of increased
inter-node communication. In contrast, the performance diﬀerence between ADWS, GET/PUT (no-cache) and
ADWS, Checkout/Checkin does not increase from single-node to multi-node executions. This is because thecache does not work eﬀectively in the Heat benchmark due to low temporal locality. Therefore, the diﬀerence
between ADWS, GET/PUT (no-cache) and ADWS, Checkout/Checkin comes only from the memory copy
overheads within each node.
Itoyori (ADWS, Checkout/Checkin) performs competitively to MPI+OpenMP in small node counts, but onlarge node counts, MPI+OpenMP outperforms Itoyori in strong scaling. The proﬁling result (Figure 7.10)
clearly shows the reason: lack of parallelism. More precisely, if parallelism is not suﬃcient, it is diﬃcult for
Itoyori to ﬁll up all cores in a short period of time. This is because of the programming model, rather than
the implementation quality. In the global fork-join model, we need to create tasks from a single root task
and then propagate tasks to workers, which takes longer latency than in MPI. In contrast, in MPI (the SPMD
model), the task assignment to processes is already determined ahead of time. The only thing they need to do
is to exchange the halo regions with their neighbors. Therefore, we consider that MPI is preferable for strong
scaling with limited parallelism.

Weak Scaling
Although Itoyori underperformed MPI+OpenMP in strong scaling, it performs competitively to MPI+OpenMPin weak scaling. Figure 7.11 and Figure 7.12 show execution time and parallel eﬃciency in weak scaling. We

99

7. Application Studies and Performance Evaluation

30

Execution time (s)Execution time (s)

2
1.5
1
0.5

20
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin

10

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin001

MPI+OpenMP

101# of nodes (× 76 cores/node)

10

100

1000

# of nodes (× 48 cores/node)(a) SQUID-C.

(b) Wisteria-O.

Figure 7.11.: Execution time of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)).

100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin40

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin20

MPI+OpenMP

0
1

10

# of nodes (× 76 cores/node)(a) SQUID-C.

1

10

100

# of nodes (× 48 cores/node)(b) Wisteria-O.

Figure 7.12.: Execution time of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)). Paralleleﬃciency of the executions in Figure 7.11 is calculated based on the best one-node execution
time among the execution policies.

1007. Application Studies and Performance Evaluation
Random WS, GET/PUT (no-cache) Random WS, Checkout/CheckinExecution time (s)1.2

ADWS, GET/PUT (no-cache)

1

1.5

0.06

0.8
1

0.1
0.04

0.6
0.4

0.5

ADWS, Checkout/Checkin
0.08

0.15

0.05

0.02

0.2
0

1

2

4

8

16

32

0

1

2

4

8

16

0

32

1

2

4

8

16

32

0

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)(a) SQUID-C

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
12
30
25

10

20

8

15

6

10

4

5

2

0

1

2

6

12 36 144 576

0ADWS, GET/PUT (no-cache)

ADWS, Checkout/Checkin
0.06

0.2

0.05
0.15

0.04

0.1

0.03
0.02

0.05

1

2

6

12 36 144 576

0

0.01
1

2

6

12 36 144 576

0

1

2

6

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

12 36 144 576# of nodes (× 48 cores/node)(b) Wisteria-O

Figure 7.13.: Proﬁling of the Heat benchmark with weak scaling ((𝑁 𝑥 , 𝑁 𝑦 ) = (32K × 𝑥, 32K)).
increased the number of elements in the x domain (𝑁 𝑥 ) proportionally to the number of nodes. As well as
strong scaling, random work stealing performs poorly due to large communication overheads. The execution
time of ADWS, GET/PUT (no-cache) is constant because caching is not important in this benchmark. The
proﬁling result (Figure 7.13) supports all these results. Overall, Itoyori (ADWS, GET/PUT (no-cache)) showscompetitive performance to MPI+OpenMP. Notably, Itoyori showed more than 95% parallel eﬃciency on 576
nodes (27,648 cores) on Wisteria-O. Summarizing, Itoyori performs very well if suﬃcient parallelism isprovided.

7.4. UTS-Mem
The memory access granularity in Cilksort and Heat is easy to enlarge by increasing the cutoﬀ count, but thisis not always the case in more dynamic and irregular workloads. UTS-Mem [151], an extension to the UTS
benchmark [84] (evaluated in Section 4.5.3), involves dynamic, irregular, and ﬁne-grained memory access.
The task of the original UTS benchmark is to count the total number of nodes in an unbalanced tree. However,
the workload of UTS is not realistic, as it does not involve global memory access. The tree is not in memory but
is dynamically generated from the root in a deterministic way, by using hash calculation during tree traversal.
In contrast, UTS-Mem generates the same tree as the original UTS and stores it in memory by allocating
memory objects from the global heap.
In our experiment, we measure the traversal time for the tree constructed in global memory in advance. As
the tree traversal is performed by chasing global pointers, many ﬁne-grained memory accesses are performed.
Although each tree node is accessed only once, runtime caching can help improve performance by exploiting
spatial data locality. In this benchmark, close tree nodes are likely to be located in close memory regions (e.g.,101Linear speedup
vs. serial

100G

Linear speedup
vs. 1 node

10G
1G
100M
1

100G

Throughput (nodes/s)

Throughput (nodes/s)

7. Application Studies and Performance Evaluation

Linear speedup
vs. serial
Linear speedup
vs. 1 node

10G

Random WS, GET/PUT (no-cache), T1L
Random WS, GET/PUT (no-cache), T1XL
Random WS, Checkout/Checkin, T1L
Random WS, Checkout/Checkin, T1XL

1G

ADWS, GET/PUT (no-cache), T1L
ADWS, GET/PUT (no-cache), T1XL

100M

10

1

# of nodes (× 76 cores/node)

10

100

ADWS, Checkout/Checkin, T1L
ADWS, Checkout/Checkin, T1XL

# of nodes (× 48 cores/node)(a) SQUID-C.

(b) Wisteria-O.

Figure 7.14.: Throughput (number of nodes visited per second) of UTS-Mem with strong scaling. The straightdashed lines represent the ideal speedup, which is calculated based on the serial execution time
and the best one-node execution time among the execution policies.

100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60

Random WS, GET/PUT (no-cache), T1L
Random WS, GET/PUT (no-cache), T1XL

40

Random WS, Checkout/Checkin, T1L
Random WS, Checkout/Checkin, T1XL

20

ADWS, GET/PUT (no-cache), T1L
ADWS, GET/PUT (no-cache), T1XL

0
1

10

# of nodes (× 76 cores/node)(a) SQUID-C.

1

10

100

ADWS, Checkout/Checkin, T1L
ADWS, Checkout/Checkin, T1XL

# of nodes (× 48 cores/node)(b) Wisteria-O.

Figure 7.15.: Parallel eﬃciency of UTS-Mem with strong scaling. Parallel eﬃciency of the executions inFigure 7.14 is calculated based on the best one-node execution time among the execution policies.

1027. Application Studies and Performance Evaluation
within the same memory block), because the tree construction is also done in parallel. During tree construction,
the memory objects for tree nodes are locally allocated with the noncollective policy (see Section 5.3.2).
Figure 7.14 shows strong scaling results for two diﬀerent tree sizes: T1L (102,181,082 nodes) and T1XL
(1,635,119,272 nodes). The y-axis shows the throughput (the number of tree nodes counted per second).
Figure 7.15 shows parallel eﬃciency in strong scaling. In this case, ADWS is not very helpful, often showing
worse performance than random work stealing. This is because we cannot predict the tree shape, which ishighly unbalanced. Overall, Itoyori scales well and greatly outperforms the no-cache version by exploiting
spatial data locality. For T1XL, Itoyori showed a good scalability (a 8.3× speedup from 12 nodes to 144 nodes
on Wisteria-O) and 7.6× better performance than the no-cache version on 144 nodes on Wisteria-O.

7.5. ExaFMM
ExaFMM [152] is a Fast Multipole Method (FMM) library for N-body simulation. It manages particles using
a tree (called an octree) by recursively partitioning the 3D space into eight parts until the number of particles
becomes less than a threshold. By leveraging the octree, it approximates the force interactions between
far enough particles to reduce computation. This makes its workload highly dynamic and irregular. On
shared memory, nested fork-join parallelization for ExaFMM has been explored [153]. This implementation
straightforwardly parallelizes the tree-based computation with a recursive divide-and-conquer method. It uses
MassiveThreads [15] for scheduling fork-join computations on multiple cores.
As a real-world case study, we ported this fork-join implementation of ExaFMM [154] to Itoyori. A
major change from the original code is the insertion of the checkout/checkin calls to where global memoryis accessed (e.g., computation kernels). In addition, we modiﬁed the program so that the parent thread stackis never accessed by its children (see Section 3.2). This is mostly accomplished by passing variables to childthreads by value. Overall, we did not have to change the whole structure of the original parallel algorithm.
Itoyori allows for much easier porting than the message-passing model, which would require redesigning theparallel algorithm.
In fact, ExaFMM has an MPI implementation [152], [154], which is a hybrid of MPI and the fork-join model
for shared memory. This implementation ﬁrst partitions the 3D domain into subdomains according to thenumber of MPI processes by using the orthogonal recursive bisection (ORB) method, so that each MPI process
has an equal number of particles. Then, each MPI process builds an octree for its local particles and computes
local particle interactions based on the local octree, exactly in the same way as the shared-memory forkjoin implementation. Overlapped with the computations for local particle interactions, it exchanges particles
and octree cells needed for computations in other MPI processes. After completing the communication for
remote data and computations for local particle interactions, it calculates interactions between local and remoteparticles, which have been received from other processes. The fork-join computations within each process arescheduled by MassiveThreads as well.
In contrast to the MPI implementation, Itoyori’s ExaFMM implementation does not require explicit domain
partitioning and communication. It builds a single global octree and computes particle interactions based on the
global octree, just as the shared-memory fork-join implementation does. Therefore, Itoyori’s implementation
is expected to be simpler and more concise. As a mere reference, we measured the lines of code (LOC) andthe diﬀerences (diﬀ) to the original shared-memory ExaFMM implementation by using the cloc command(v1.90) [155]. To extract the essential code from the entire source code, we followed the same procedure asin Section 7.2. For the kernel code, we only included the Laplace kernel. Table 7.2 shows the results for the103

7. Application Studies and Performance Evaluation

Table 7.2.: Lines of code (LOC) summary for diﬀerent implementations of ExaFMM. The diﬀ (modiﬁed, added,
and removed lines) to the original shared-memory ExaFMM implementation (MassiveThreads) was
calculated by the cloc command.
Implementation

MassiveThreads
MPI+MassiveThreads
Itoyori

LOC (addition to MassiveThreads)

881 LOC
1436 LOC (+555 LOC)
1045 LOC (+164 LOC)

diﬀ to MassiveThreads
modiﬁedadded

removed

7
245

555
239

0
75

MPI implementation and the Itoyori Implementation. The MPI implementation adds 555 LOC to the originalcode, with little modiﬁcation to the original shared-memory computation parts. This addition is mainly fordomain partitioning (ORB) and exchange for particles and tree cells using explicit MPI communication. On
the other hand, Itoyori adds only 164 LOC to the original shared-memory fork-join code. This addition ismainly for additional checkout/checkin APIs. Nevertheless, while the overall increase in LOC is small, there
is a considerable amount of diﬀ. These changes are mainly for changing types of pointers, references, andvector containers (see Itoyori’s high-level C++ APIs in Section 3.4). However, most of these changes can bemade procedurally, without altering the logic of the algorithm. In contrast, the MPI implementation requires a
new parallelization logic, which entails additional costs in algorithm development and result veriﬁcation, thus
involving complexity beyond just the number of LOC.
In our experiments, we computed the Laplace kernel with the parameters 𝜃 = 0.5, 𝑁crit = 32, 𝑃 = 5,
nspawn = 1000 (see [153] for these parameters), with dual tree traversal enabled. We evaluated using two
types of particle distribution: homogeneous distribution in a 3D cubic space (cube) and biased distributionthat follows the Plummer model [156] (plummer). The plummer model represents the distribution of globularclusters and often used in N-body simulations. Roughly speaking, it forms a spherical distribution where theparticle density is higher near the center and lower in the outer areas. We spawned one MPI process per nodeand launched as many MassiveThreads workers as the number of cores in each node. We report only the time
for computing particle interactions, excluding tree build and partitioning time.

Strong Scaling
Figure 7.16 and Figure 7.17 show the strong scaling results (execution time and parallel eﬃciency) for 10M
particles. They compare various execution policies with two types of particle distribution. Technically, the
GET/PUT (no-cache) implementation is illegal in C++ because each octree cell has nontrivially copyable,
global vector containers (see Section 5.2). However, since they were executed without any issues, we haveplotted the performance for reference.
Overall, the cache-enabled versions outperformed the no-cache version (up to 9.8× faster on SQUID-C).
This large performance improvement is due to ExaFMM’s high degree of temporal and spatial locality forboth particles and octree cells, although the computation pattern is irregular. Notably, Itoyori (ADWS,
Checkout/Checkin) exhibited a 10.5× speedup when scaled from a single node to 12 nodes on Wisteria-O forthe cube distribution. Even compared with MPI+MassiveThreads, it was a 9.6× speedup. This corresponds
to a 460× speedup and 80% parallel eﬃciency (calculated based on the serial execution time) on 576 cores.
Still, MPI+MassiveThreads was 1.2× faster than Itoyori on this number of nodes.
The single-node performance of Itoyori was about 1.1× slower than MassiveThreads on both SQUID-C and104

Execution time (s)Execution time (s)7. Application Studies and Performance Evaluation

10

1

10

1

cube

plummer
101(a) SQUID-C, cube.

(b) SQUID-C, plummer.

10

1

10

# of nodes (× 76 cores/node)

Execution time (s)Execution time (s)1# of nodes (× 76 cores/node)

1

10

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

10

1

cube

MPI+MassiveThreads

plummer100

1# of nodes (× 48 cores/node)

10

100

# of nodes (× 48 cores/node)(c) Wisteria-O, cube.

(d) Wisteria-O, plummer.

Figure 7.16.: Execution time of ExaFMM (particle interactions) with strong scaling. The straight dashed lines
represent the ideal speedup (calculated based on the serial execution time).
100
cube
80
60
40
20

Parallel Efficiency (%)

Parallel Efficiency (%)

100

0plummer
80
60
40
20
0

1

10

1

# of nodes (× 76 cores/node)(a) SQUID-C, cube.

80
60
40
20
0

Parallel Efficiency (%)

100
cube

MPI+MassiveThreads
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin(b) SQUID-C, plummer.

100

Parallel Efficiency (%)

10# of nodes (× 76 cores/node)

plummer80

ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

60
40
20
0

1

10

100

# of nodes (× 48 cores/node)(c) Wisteria-O, cube.

1

10

100

# of nodes (× 48 cores/node)(d) Wisteria-O, plummer.

Figure 7.17.: Parallel eﬃciency of ExaFMM (particle interactions) with strong scaling. Parallel eﬃciency ofthe executions in Figure 7.16 is calculated based on the serial execution time.
105

Time ratio

7. Application Studies and Performance Evaluation

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

32

0

1

2

4

8

16

32

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

32

1

0

ADWS, Checkout/Checkin

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, cube.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

32

0

1

2

4

8

16

32

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

32

1

0

ADWS, Checkout/Checkin

1

2

4

8

16

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program32# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, plummer.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36 144

0

1

2

6

12

36 1440

ADWS, GET/PUT (no-cache)

1

2

6

12

36 1441

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

36 144# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, cube.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36 144

0

1

2

6

12

36 1440

ADWS, GET/PUT (no-cache)

1

2

6

12

36 1441

0

ADWS, Checkout/Checkin

1

2

6

12

36 144# of nodes (× 48 cores/node)(d) Wisteria-O, plummer.

Figure 7.18.: Proﬁling of ExaFMM (particle interactions) with strong scaling.

106

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program7. Application Studies and Performance Evaluation
Wisteria-O. The proﬁling result (Figure 7.18) shows why. On single-node executions, the Checkout/Checkinpolicies have 10%–20% overhead in checkout/checkin calls. This indicates that checkout/checkin calls are
frequent and sequential overheads for these calls are not negligible. In ExaFMM, checkout/checkin operations
are made for each octree cell during tree traversal, and octree nodes are created until the number of particles
becomes less than 𝑁crit = 32. This results in ﬁne-grained access to octree cells.
For the cube distribution, while MPI often slightly outperforms, the performance of Itoyori and MPI iscompetitive. On the other hand, MPI performed poorly with the plummer distribution. This is because ofload imbalance caused by biased particle distribution. In such a biased distribution like plummer, even withan equal number of particles, load imbalance can occur. This is because particles in regions of high density
have more neighboring particles, requiring more particle interactions. As the MPI implementation performs
domain partitioning based on the number of particles and does not perform dynamic load balancing after that,it suﬀered from load imbalance.

Weak Scaling
The proﬁling result for strong scaling (Figure 7.18) shows that parallelism is not suﬃcient on large node counts.
Then, we conducted weak scaling experiments for ExaFMM. For weak scaling, we increased the number ofparticles proportional to the number of nodes. At this time, the number of nodes is set to a power of eight(23 ), which is also the practice in [152]. This is because the amount of computation of ExaFMM depends on
the number of octree cells, rather than on the number of particles. As the tree is an octree, the octree depth isexpected to increment when the number of particles increases by a factor of eight.
Figure 7.19 and Figure 7.20 shows execution time and parallel eﬃciency in weak scaling. We could not
execute on 64 nodes and more on SQUID-C because of too long communication latency (in the order of
seconds), which randomly happened during the execution of Itoyori. MPI+MassiveThreads also failed to run
on 64 nodes on Wisteria-O for the plummer distribution because of the out-of-memory error. We found that
the MPI implementation consumed too much memory for data buﬀers for exchanging particles and cells forplummer. We consider that this is because of the biased distribution of plummer; particular nodes need a hugeamount of communication.
Even with weak scaling, both MPI and Itoyori were not very scalable for the plummer distribution. An
interesting ﬁnding is that random work stealing performed better than ADWS for plummer. Figure 7.21 shows
the proﬁling result for weak scaling, which indicates that ADWS incurred larger scheduler loop time. Thisis because of the dynamic load balancing scheme in ADWS; it dynamically limits the steal range to speciﬁcworkers. Therefore, when the computation is highly irregular and imbalanced, ADWS fails to ﬁnd available
tasks in a short period of time.
In contrast, for the cube distribution, both MPI and Itoyori show high scalability to 512 nodes. Some
conﬁgurations of Itoyori could not run on 512 nodes because of large memory consumption for page tables
managed by the OS. On par, MPI+MassiveThreads was about 1.1 to 1.2× faster for the cube distribution.
Figure 7.21 shows that this performance diﬀerence can be mainly attributed to the sequential overheads forcheckout/checkin calls. Notably, Itoyori maintained 75% parallel eﬃciency even on 512 nodes. In addition,
ADWS performed better than random work stealing by 1.1× on 64 nodes. Figure 7.21 shows that the checkout
time was reduced from random work stealing, which means that the data locality was improved by ADWS.
Here, ADWS is eﬀective because the particle distribution is homogeneous and the computation is well balanced
for cube.
Overall, particle interactions of ExaFMM is compute-bound and communication is unlikely to be a major

107

7. Application Studies and Performance Evaluation

Execution time (s)Execution time (s)

50

cube

80
60
40
20

plummer

40
30
20
10

0

0
1

1

# of nodes (× 76 cores/node)# of nodes (× 76 cores/node)(a) SQUID-C, cube.

(b) SQUID-C, plummer.
200

150

Execution time (s)Execution time (s)

cube

100

50
1

10

plummer

MPI+MassiveThreads
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

150
100
50

100

1

# of nodes (× 48 cores/node)

10# of nodes (× 48 cores/node)(c) Wisteria-O, cube.

(d) Wisteria-O, plummer.

Figure 7.19.: Execution time of ExaFMM (particle interactions) with weak scaling.
100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
cube
0

80
60
40
20
plummer01

1# of nodes (× 76 cores/node)# of nodes (× 76 cores/node)(a) SQUID-C, cube.

(b) SQUID-C, plummer.
100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
cube
0

MPI+MassiveThreads
Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

80
60
40
20
plummer
0

1

10

100

# of nodes (× 48 cores/node)(c) Wisteria-O, cube.

110# of nodes (× 48 cores/node)(d) Wisteria-O, plummer.

Figure 7.20.: Parallel eﬃciency of ExaFMM (particle interactions) with weak scaling. Parallel eﬃciency ofthe executions in Figure 7.19 is calculated based on the serial execution time.
108

7. Application Studies and Performance Evaluation

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
80
60

ADWS, Checkout/Checkin

10

8

8

8

6

6

4

4

2

2

6
40

ADWS, GET/PUT (no-cache)

4

20

2

0

1

0

8

1

0

8

1

0

8

1

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program8# of nodes (× 76 cores/node)(a) SQUID-C, cube.

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
50
40
30

ADWS, GET/PUT (no-cache)

8

40

6

30

4

20

2

10

0

1

0

8

1

6
4
2

0

8

ADWS, Checkout/Checkin

8

20
10

10

1

0

8

1

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program8# of nodes (× 76 cores/node)(b) SQUID-C, plummer.

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
80
150

60

100

40

50

20

0

0

1

8

64

1

8

64

ADWS, GET/PUT (no-cache)

ADWS, Checkout/Checkin

60

60

50

50

40

40

30

30

20

20

10

10

0

1

8

64

512

0

1

8

64

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program512# of nodes (× 48 cores/node)(c) Wisteria-O, cube.
Random WS, GET/PUT (no-cache) Random WS, Checkout/CheckinExecution time (s)120ADWS, GET/PUT (no-cache)
200

50

ADWS, Checkout/Checkin
80

100
40

80

30

60
40

20

20

10

0

1

8

64

0

1

8

64

150

60

100

40

50

20

0

1

8

64

0

1

8

64

# of nodes (× 48 cores/node)(d) Wisteria-O, plummer.

Figure 7.21.: Proﬁling of ExaFMM (particle interactions) with weak scaling.

109

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program7. Application Studies and Performance Evaluation
bottleneck if remote memory access is appropriately cached by the runtime system. More importantly,
parallelism plays a key role in scaling ExaFMM to a large number of nodes; the critical path length can belong depending on the particle distribution.

7.6. ANNS Graph Construction
𝑘-nearest neighbor search is a query for the top-𝑘 vectors that are the closest to a given vector. In contrast,
Approximate Nearest Neighbor Search (ANNS) is a query for top-𝑘 vectors that are close enough to a given
vector, which does not require exact solutions. Many ANNS algorithms have been investigated until now,and recently, graph-based ANNS has attracted attention because of its good trade-oﬀ between accuracy andeﬃciency. In graph-based ANNS, close vectors are connected with edges, and a query traverses the edges
while calculating distances to the given point. Recently, Dobson et al. [157] summarized parallel implementations for graph construction algorithms for ANNS. As their C++ implementations are based on the fork-joinmodel and written in a data-race-free manner, they can be easily ported to Itoyori. Therefore, we picked up
the implementations from their GitHub repository https://github.com/cmuparlay/pbbsbench-vldb2024(commit hash: 41ca799915f6) and ported two ANNS algorithms to Itoyori: Hierarchical Navigable Small
World Graphs (HNSW) [158] and Hierarchical-Clustering-based Nearest Neighbor Graph (HCNNG) [159].
The major diﬀerence between HNSW and HCNNG is how a graph is constructed. Graph constructionfor HNSW is an incremental approach, in which new data points are inserted to nearby points, which aresearched by traversing the previous graph. In their parallel algorithm, multiple data points are batchedand simultaneously inserted in parallel. The batch size is increased in a power of two. In contrast, graphconstruction for HCNNG is based on a divide-and-conquer approach, in which data points are recursively
partitioned into two subgroups by calculating distances to randomly selected two points. At leaf groups at agiven threshold, the points in each leaf group are assigned edges corresponding to a minimum spanning tree
(MST) over them. This procedure is repeated for multiple times.
We ported these two ANNS implementations to Itoyori. This porting task was straightforward because thesehave already written in a data-race-free manner. For HNSW, parallelism exists only for the batched data points;
it can be written in a simple parallel for loop. For the divide-and-conquer graph construction in HCNNG, we
can recursively parallelize two subtasks by fork-join calls. To manage a global graph structure, they manage
global edge lists for each data point. They update the edge lists when edges are added to or removed fromdata points. As these operations involve checkout/checkin calls in Itoyori, checkout/checkin calls tend to be
ﬁne-grained in these applications.
For evaluation, we used two datasets for experiments: BIGANN (bigann-10M) and Microsoft SPACEV
Dataset (msspacev-10M) with 10M data points. As they exhibited similar performance trends, in this section we
only show the results of bigann-10M. We show the results of msspacev-10M in Section B.1. The datasets wereobtained by using the script in the big-ann-benchmarks dataset repository [160]. As the graph construction
took a considerable amount of time, we plotted the median of three runs in the following graphs.
Figure 7.22 shows graph construction time, and Figure 7.23 shows parallel eﬃciency in strong scaling.
For the Itoyori (with checkout/checkin) executions on 36 nodes on Wisteria-O, the executions of HNSW
were somehow killed by the system job manager (unknown reason), so they are omitted from the graphs.
These results clearly show that the scalability of HNSW is poor, while that of HCNNG is relatively good.
This is because HNSW requires a graph search operation for each data point to be added. Therefore, tasks
that are close in the computation graph are unlikely to share data, showing poor data locality. The proﬁling

110

7. Application Studies and Performance Evaluation

1000

Execution time (s)Execution time (s)

1000

100

100

10

10
1

10

1

# of nodes (× 76 cores/node)(b) SQUID-C, HCNNG.

Execution time (s)Execution time (s)(a) SQUID-C, HNSW.

1000

100

1

10

# of nodes (× 76 cores/node)Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

1000

100

10

1

# of nodes (× 48 cores/node)

10# of nodes (× 48 cores/node)(c) Wisteria-O, HNSW.

(d) Wisteria-O, HCNNG.

Figure 7.22.: Execution time of ANNS graph construction with strong scaling (bigann-10M). The straightdashed lines represent the ideal speedup (calculated based on the serial execution time).
100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60
40
20
0

1

10

1

# of nodes (× 76 cores/node)(a) SQUID-C, HNSW.

(b) SQUID-C, HCNNG.
100

Parallel Efficiency (%)

100

Parallel Efficiency (%)

10# of nodes (× 76 cores/node)

80
60
40
20
0

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

80
60
40
20
0

1

10

# of nodes (× 48 cores/node)(c) Wisteria-O, HNSW.

1

10# of nodes (× 48 cores/node)(d) Wisteria-O, HCNNG.

Figure 7.23.: Parallel eﬃciency of ANNS graph construction with strong scaling (bigann-10M). Parallel eﬃciency of the executions in Figure 7.16 is calculated based on the best one-node execution time
among the execution policies.
111

Time ratio

7. Application Studies and Performance Evaluation

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, HNSW.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, HCNNG.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program

12# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, HNSW.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

36

# of nodes (× 48 cores/node)(d) Wisteria-O, HCNNG.

Figure 7.24.: Proﬁling of ANNS graph construction with strong scaling (bigann-10M).

112

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser ProgramQuery Per Second (QPS)

Query Per Second (QPS)

7. Application Studies and Performance Evaluation
1M

100k

HCNNG (Itoyori)
HCNNG (PBBS)HNSW (Itoyori)
HNSW (PBBS)

10k
0.0

0.2

0.4

0.6

Recall 10@10

0.8

1.0

100k

HNSW (Itoyori)
HNSW (PBBS)
0.0(a) SQUID-C (76 cores).

HCNNG (Itoyori)
HCNNG (PBBS)

10k

0.2

0.4

0.6

Recall 10@10

0.8

1.0

(b) Wisteria-O (48 cores).

Figure 7.25.: Query per second (QPS) on one node with various recall rates (bigann-10M). Shown to checkthe validity of the graph obtained as a result.
results in Figure 7.24 show that most of the execution time of HNSW was for communication on multiple
nodes for all execution policies. Even worse, the performance of Checkout/Checkin was much worse than
GET/PUT (no-cache) on Wisteria-O. This is because of the mmap() cost needed to manage virtual memory
mappings for cache blocks. In contrast, for HNSW on SQUID-C, the performance of Checkout/Checkin is
not so diﬀerent from GET/PUT (no-cache). This is because of the balance between the mmap() cost andcommunication latency. On Wisteria-O, the mmap() latency was in the order of ten microseconds, while
the communication latency was in the order of microseconds. In contrast, on SQUID-C, the latency for both
mmap() and communication was in the order of microseconds. As explained in Section 5.3.3, Itoyori’s cache
implementation tries to hide the mmap() cost by overlapping it with communication. However, on Wisteria-O,
the mmap() latency was much larger than the communication latency, resulting in large cache miss penalty.
In contrast, HCNNG showed 6.6× speedup on 12 nodes on Wisteria-O, which is about 55% paralleleﬃciency compared to the single-node execution. Figure 7.24 indicates a lack of parallelism on large nodecounts for Itoyori with checkout/checkin calls. Experiments with larger dataset are our future work.
To check the validity of the graph obtained as a result, we show query per second (QPS) with various recall
rates in Figure 7.25. “PBBS” is the original parallel implementation of HNSW and HCNNG by Dobson etal. [157]. We executed the QPS measurement on a single node by using all cores within the node. Although
there are slight diﬀerences in QPS in Itoyori and PBBS, they show similar performance trends. This suggeststhat their implementation was successfully ported to Itoyori. In fact, veriﬁcation of Itoyori is much easier thanother distributed-memory implementations because the graph can be built in a deterministic way, regardlessof the number of nodes used for graph construction. This is because the Itoyori program exposes only logicalparallelism in a processor-oblivious way.
Summarizing, experiments with ANNS graph construction with two methods (HNSW and HCNNG) haveshown interesting performance trends. Although it is easy for Itoyori to port existing shared-memory parallel
programs to distributed memory, their performance depends on the degree of data locality and parallelism they
expose. For example, HNSW requires a full graph search for each data point to be added, which results inpoor data locality. In contrast, HCNNG employs a divide-and-conquer strategy for graph construction, whichleads to better data locality than HCNNG. Like these examples, Itoyori users are required to write programs ina cache-friendly way to achieve high scalability and eﬃciency.

7.7. PageRank113

7. Application Studies and Performance Evaluation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

int N = /* the number of vertices */;ityr::global_span<vertex_data> vertices(/* ... */); // size N
ityr::global_span<double> PR(/* ... */);// size N
ityr::global_span<double> PR_next(/* ... */);// size N/* ... */while (/* termination condition */) {// for each vertex 0..N-1
ityr::for_each(
ityr::execution::par,ityr::count_iterator(0), ityr::count_iterator(N),[=](int v_id) {vertex_data v = vertices[v_id].get(); // Get the vertex data of v// calculate sum of PR values for each neighbor of v
double contribution =ityr::transform_reduce(
ityr::execution::par,v.neighbors().begin(), v.neighbors().end(),ityr::reducer::plus<double>{},[=](int u_id) {return PR[u_id].get(); // Get the current PR of u (neighbor of v)});
PR_next[v_id].put(contribution * /* ... */); // Put the next PR of v});
std::swap(PR, PR_next);}

Figure 7.26.: Naive PageRank implementation in Itoyori.
PageRank is a graph algorithm to compute an importance of each vertex in the graph. For example, vertices
that have many incoming edges are important, and vertices linked by important vertices are also important. It
is often used for web search engines by representing pages as vertices and edges as links between pages.
In PageRank, each vertex has its own PageRank value, and the PageRank values are propagated throughedges. PageRank is an iterative algorithm; it repeats updating the current PR values by accumulating PR values
from their neighbors in one iteration. In other words, PageRank can be considered to be a stencil computationin graphs. The PageRank algorithm is terminated when PR values become stable.
Figure 7.26 shows a naive implementation of PageRank written with Itoyori. This code shows a doubly
nested parallel loop: the outer loop iterating over all vertices and the inner loop iterating over the neighbors for each vertex. The neighbors of vertices are internally stored in the compressed sparse row (CSR)
format. The vertex data struct has a global pointer to the neighbor list and the number of neighbors. The
neighbor() method of vertex data returns a global span for the neighbor list array. For the inner loop, theityr::transform reduce() function can be used to accumulate the PR values of the neighbors (reduction
with the + operator). However, the inner loop exhibits poor data locality; it causes random memory accessto the PR array. Note that this performance issue is not speciﬁc to distributed memory. Even on sharedmemory, random memory access can be harmful for hardware caches. In fact, this situation is the sameas sparse matrix-vector multiplication (SpMV). Nevertheless, many graph processing frameworks, including
Graph Based Benchmark Suite (GBBS) [161], [162], employs this parallel algorithm.
To improve data locality of PageRank (and SpMV), many cache-friendly algorithms have been investigated.
Partition-Centric Processing Methodology (PCPM) proposed by Lakhotia et al. [163] is one of the cachefriendly PageRank algorithms for shared memory. It preprocesses the graph data structure so that the vertices

114

7. Application Studies and Performance Evaluation
are divided into multiple equal-sized partitions. PR values of partition 𝑖 are propagated to partition 𝑗 througha bin (bin[𝑖] [ 𝑗]). Conceptually, the adjacency matrix is divided into smaller square grids, and each gridcorresponds to a bin. Partition 𝑖 ﬁrst puts PR values of local vertices to bin[𝑖] [ 𝑗] (the scatter phase). Here,partition 𝑗 is determined based on each neighbor vertex ID. After the scatter phase is completed for allpartitions, partition 𝑖 accumulates PR values stored in bin[𝑘] [𝑖] for all 𝑘 (the gather phase). This method
is cache-friendly because of its eﬃcient bin implementation (see [163] for more details). Later, PCPM was
generalized to support various kinds of graph algorithms and integrated into the Graph Processing Over Parts(GPOP) framework [164]. In the following, we simply call this method “GPOP.”For evaluation of Itoyori, we ported GPOP to distributed memory with Itoyori APIs. Because of the
preprocessing of the graph data structure, course-grained checkout/checkin requests are possible for the datain each bin. Although the original GPOP implementation is parallelized with a single parallel loop overpartitions, but we found that this policy severely limited parallelism, particularly for graphs with highly skewed
degree distribution. Therefore, we applied ﬁne-grained parallelization to the inner loops in GPOP scatter andgather phases.
In our evaluation, we used two graph datasets: R-MAT graph (rmat) [165] (synthetic graph) and twitter2010
graph [166]. twitter2010 has 𝑁 = 41, 652, 230 vertices and 𝑀 = 1, 468, 364, 884 edges. Web graphs liketwitter2010 often have a highly skewed degree distribution, and R-MAT graphs are synthetically created to
mimic the skewed degree distribution found in real-world graphs. For generating R-MAT graphs, we usedparameters 𝑎 = 0.57, 𝑏 = 0.19, 𝑐 = 0.19 and created 16 times the number of edges as vertices. For example,
for the rmat26 graph, we allocate 𝑁 = 226 vertices and 𝑀 = 230 edges. For GPOP, we set the partition size as
32K. In the following ﬁgures, we plotted execution time of three iterations of PageRank.

Strong Scaling
Figure 7.27 shows execution time of various PageRank algorithms and execution policies with strong scalingon multiple nodes. Nevertheless, none of the implementations scales very well. The naive implementationof PageRank with Itoyori clearly shows poor performance regardless of execution policies. In fact, mostof them underperformed the serial GPOP performance. For the GPOP implementation with Itoyori, execution policies with checkout/checkin calls performed better than others. However, they never signiﬁcantly
exceeded the performance on a single node. For reference, we also measured the performance of PageRank in
GBBS [161], [162], which is a naive (cache-unfriendly) PageRank implementation for shared memory. GBBS
underperformed GPOP on a single node because of its poor locality.
In fact, eﬃcient and scalable graph processing systems are challenging on distributed memory in general.
Previously, McSherry et al. [167] pointed out that many distributed graph processing systems were ineﬃcientand sometimes even slower than a highly optimized single-thread implementation. To this end, Gemini [168]has been developed as a highly eﬃcient and scalable distributed graph processing system. We also executed
PageRank with Gemini and plotted its results in Figure 7.27. Note that Gemini could not run on Wisteria-O
because MPI THREAD MULTIPLE was not supported by Fujitsu MPI. However, Gemini was not so scalable too,
although Gemini and GPOP (with Itoyori) are in the same performance range. Gemini’s paper [168] reported
good scalability of PageRank to 8 nodes for the twitter2010 dataset, but at that time each node has only 24
cores. In our environment, each node has as many as 76 cores. We consider that the intra-node performanceis suﬃciently high today, leaving little room for performance improvements on multiple nodes.

115

7. Application Studies and Performance Evaluation

Itoyori-Naive, Random WS, GET/PUT (no-cache)
Itoyori-Naive, Random WS, Checkout/Checkin
Itoyori-Naive, ADWS, GET/PUT (no-cache)
Itoyori-Naive, ADWS, Checkout/Checkin

Itoyori-GPOP, Random WS, GET/PUT (no-cache)
Itoyori-GPOP, Random WS, Checkout/Checkin
Itoyori-GPOP, ADWS, GET/PUT (no-cache)
Itoyori-GPOP, ADWS, Checkout/Checkin

Gemini (MPI+OpenMP)
GBBS

Serial (Naive)

10

Execution time (s)Execution time (s)100

Serial (GPOP)

1

GBBS (1 node)

1

Serial (Naive)

10

1
GBBS (1 node)

10

1# of nodes (× 76 cores/node)(b) SQUID-C, twitter2010.
Serial (Naive)

Execution time (s)Serial (Naive)

Execution time (s)10# of nodes (× 76 cores/node)(a) SQUID-C, rmat26.

100
Serial (GPOP)

10

Serial (GPOP)

GBBS (1 node)

1
1

10

100
Serial (GPOP)

10

GBBS (1 node)

1
1# of nodes (× 48 cores/node)

10# of nodes (× 48 cores/node)(c) Wisteria-O, rmat26.

(d) Wisteria-O, twitter2010.

Figure 7.27.: Execution time of PageRank (three iterations) with strong scaling.

116

7. Application Studies and Performance Evaluation

Itoyori-Naive, Random WS, GET/PUT (no-cache)
Itoyori-Naive, Random WS, Checkout/Checkin
Itoyori-Naive, ADWS, GET/PUT (no-cache)
Itoyori-Naive, ADWS, Checkout/Checkin

Itoyori-GPOP, Random WS, GET/PUT (no-cache)
Itoyori-GPOP, Random WS, Checkout/Checkin
Itoyori-GPOP, ADWS, GET/PUT (no-cache)
Itoyori-GPOP, ADWS, Checkout/Checkin

Gemini (MPI+OpenMP)
GBBS

100

Execution time (s)Execution time (s)

1000

10
1

100
10
1

0.1
0.1
1

10

1

# of nodes (× 76 cores/node)

10# of nodes (× 48 cores/node)(a) SQUID-C.

(b) Wisteria-O.

Figure 7.28.: Execution time of PageRank (three iterations) with weak scaling (R-MAT: 𝑁 = 224 ×𝑥, 𝑀 = 16𝑁).

Weak Scaling
We also conducted weak scaling experiments with R-MAT graphs (𝑁 = 224 × 𝑥, 𝑀 = 16𝑁, where 𝑥 is thenumber of nodes). Figure 7.28 show the result of weak scaling. The execution times of Itoyori-GPOP scaled
along with the 𝑦 = 𝑎𝑥 line, although we expect the execution time to be constant in weak scaling. Gemini’s
performance is also on this line. This means that there is almost no beneﬁt in using multiple nodes, except forthe cases where the graph data do not ﬁt into the single-node memory.
Figure 7.29 shows proﬁling results with strong scaling for the twitter2010 dataset, and Figure 7.30 shows
proﬁling results with weak scaling. Itoyori-GPOP (ADWS, Checkout/Checkin) in Figure 7.29 indicates thatparallelism is not enough, but that in Figure 7.30 suggests that most of the execution time was consumed forcommunication.
While no beneﬁts of distributed memory were observed in PageRank, the results have many implications.
First, porting shared-memory algorithms to Itoyori is easy, but the performance is not guaranteed. If thealgorithm does not exhibit suﬃcient locality, Itoyori performs poorly. In order to improve performance of
Itoyori, the programmer needs to write cache-friendly algorithms. On the positive side, such cache-friendly
algorithm development lies in the extension of research on shared memory. For example, GPOP was originallydeveloped for shared memory, but this algorithm could improve performance on distributed memory with
Itoyori. As a result, Itoyori-GPOP achieved close performance to Gemini, an existing distributed graphprocessing system designed for eﬃciency and scalability.

117

Time ratio

7. Application Studies and Performance Evaluation

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, Itoyori-Naive.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, Itoyori-GPOP.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program36# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, Itoyori-Naive.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program36# of nodes (× 48 cores/node)(d) Wisteria-O, Itoyori-GPOP.

Figure 7.29.: Proﬁling of PageRank (three iterations) with strong scaling (twitter2010).

118

7. Application Studies and Performance Evaluation

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin

ADWS, GET/PUT (no-cache)

Execution time (s)

150

200

150

150
150

100

100

50

0

ADWS, Checkout/Checkin

100

50

1

2

4

8

16

0

100

50

1

2

4

8

16

0

50

1

2

4

8

16

0

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)(a) SQUID-C, Itoyori-Naive.
Random WS, GET/PUT (no-cache) Random WS, Checkout/CheckinExecution time (s)

200
1.5

150

1

100

0.5

50
0

1

2

4

8

16

0

1

2

4

8

16

ADWS, GET/PUT (no-cache)
250

2.5

200

2

150

1.5

100

1

50

0.5

0

1

2

4

8

16

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)(b) SQUID-C, Itoyori-GPOP.

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
200
400
300

100

200

50

100

1

2

4

8

16

0

ADWS, Checkout/Checkin
400

150

150

0

ADWS, GET/PUT (no-cache)

300
100
200
50

1

2

4

8

16

0

100

1

2

4

8

16

0

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 48 cores/node)(c) Wisteria-O, Itoyori-Naive.

Execution time (s)Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin

8

8
400

400

6

6
4

200

4

200

2

2
0

1

2

4

8

16

0

ADWS, Checkout/Checkin
10

600

10

600

ADWS, GET/PUT (no-cache)

1

2

4

8

16

0

1

2

4

8

16

0

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 48 cores/node)(d) Wisteria-O, Itoyori-GPOP.

Figure 7.30.: Proﬁling of PageRank (three iterations) with weak scaling (R-MAT: 𝑁 = 224 × 𝑥, 𝑀 = 16𝑁).

119

8. Related Work8.1. Distributed Work Stealing
Child stealing on distributed memory has been intensively studied. Dinan et al. [55] implemented RDMA-based
work stealing in the Scioto infrastructure [54] with an eﬃcient implementation of the steal-half queue [169].
Structured Atomic Work Stealing (SAWS) [56] further improved the queue implementation by utilizing RDMA
atomic operations. The global load balancing (GLB) library [79], [87] was developed for the X10 language
using lifeline-based work stealing [83]. The above systems are all based on a BoT, in which task dependencies
cannot be described.
Satin [91] and HotSLAW [58] support fork-join parallelism with RtC threads, so they are subject to the“buried join” problem [89] explained in Section 4.4.2. Grappa [59] also uses RtC threads, but it mitigates theburied join problem by creating many user-level workers (rather than threads) as Full threads to oversubscribe
cores. This approach is essentially similar to that of our child stealing implementation with Full threads tied
to each core. KAAPI [57] uses an approach similar to that of Grappa by using kernel-level threads. These
implementations are tied-tasks; i.e., once a task begins, it cannot be migrated to other nodes.
Compared with child stealing, continuation stealing are rarely implemented on distributed memory. Distributed Cilk [60] and Tascell [14] support continuation stealing but they require special compiler support.
Charm++ [85], [86] and Adaptive MPI [93] support thread migration on distributed memory based on the
iso-address scheme [92], but they are not designed for RDMA. MassiveThreads/DM (uni-address threads) [28],[29] is the only library that supports RDMA-based continuation stealing (as explained in Section 2.2.5).
Many researchers have investigated topology-aware work stealing to reduce the number of remote workstealing requests [78], [106], [107], [131], [170], [171], but they basically assumed two-sided communication.
Although these approaches could be used in conjugation with RDMA-based work stealing, their beneﬁts havenot been well studied in the context of RDMA, which is our future interest.
Performance comparison between child and continuation stealing on shared memory has also been conducted [43], [49], [172], and the tradeoﬀs between them have been reported. They found that, roughly speaking,child stealing is preferable for shallow parallel loops in which a parent task creates many child tasks, whereas
continuation stealing is preferable for deeply nested parallelism, which is consistent with our results.

8.2. Global Address Space and Software Caching
The concept of a global address space is old; the research has begun in the form of distributed sharedmemory (DSM) systems, such as IVY [173], Munin [174], TreadMarks [61], [62], and Midway [63]. DAG
consistency [67], [68] for fork-join parallelism was also explored in 1990s. Recent advances in RDMAcapable network interconnects have motivated researchers to investigate RDMA-friendly DSM systems, suchas ArgoDSM [37], Popcorn [175], and MENPS [64]. In these DSM systems, transparency is achieved bytrapping memory protection faults at the page granularity, but it comes at a cost.

120

8. Related WorkAs discussed in Section 2.3, the PGAS model has attracted attention because of its performance and programmability. Most PGAS systems prefer explicit communication and thus do not have a caching mechanism,
but exceptionally, some PGAS systems [36], [118]–[121] implement a software cache. However, they assume
the SPMD model and do not target the global fork-join model, in which tasks are dynamically scheduled acrossnodes. In addition, they implement a software cache over the GET/PUT APIs, which fall short for our purpose
as discussed in Section 5.2.
While some PGAS systems support inter-node dynamic load balancing for tasks [54], [58], [59], [78],[79], a majority of “asynchronous” PGAS systems support only intra-node dynamic load balancing, including
Chapel [16], HPX [76], GMT [176], HabaneroUPC++ [52], AsyncSHMEM [53], X10 [17], and tasking
extensions to Dash [177] and XcalableMP [178]. In these systems, inter-node load balancing is still on theuser’s responsibility. Their intent of intra-node dynamic task scheduling is partly to overlap communicationand computation by context switching to other tasks while communication is ongoing, which is orthogonal
to our caching approach. Grappa [59], [151] is a system that takes this idea into account while supportinginter-node work stealing, but without software caches. The idea of communication-computation overlap may
also be applicable to Itoyori, but we leave this investigation for future work.
Another spectrum of research is task-based distributed runtime systems that tightly couple tasks and data toperform inter-node dynamic load balancing. These systems include Legion [179], KAAPI [57], PaRSEC [180],
StarPU [181], OmpSs@Cluster [182], [183], and Tascell [14]. In these systems, the input/output data of eachtask are implicitly or explicitly speciﬁed by users with access privileges, and the data are automatically moved
to the nodes where the associated tasks are executed. Itoyori is diﬀerent from these systems in that it decouples
data from the task speciﬁcations for simplicity of APIs and ﬂexibility of global memory access.
Compiling shared-memory programs to distributed-memory programs is an alternative approach to achieving
better productivity on distributed memory. Compilation from regular, loop-based OpenMP programs to MPI
programs has been demonstrated through data-ﬂow analysis [184], [185]. However, this approach is notapplicable to dynamic and irregular applications that our work focuses on.

8.3. Locality-Aware Task Scheduling
Conventional random work stealing [26] (explained in Section 2.2.2) has been shown to exploit hierarchical data
locality well on private caches [27], but not on shared caches. The parallel depth-ﬁrst (PDF) scheduler [51],[186] was one of the earliest schedulers tailored for shared caches. The idea is that all workers simultaneously
execute tasks that are as close as possible in depth-ﬁrst order, i.e., an order similar to that of serial execution.
Variants of PDF schedulers, such as AsyncDF [187], have also been proposed to reduce the overhead with ﬁnegrained parallelism. An experimental study by Chen et al. [50] demonstrated that the PDF scheduler performs
better than work stealing on a shared cache. However, the PDF scheduler cannot well exploit hierarchical data
locality well on private caches, which has led to research on schedulers for both private caches and a sharedcache. Narlikar [188] proposed a scheduler that combined work stealing with the PDF scheduler, and Blelloch
et al. [132] proposed the Controlled-PDF scheduler. The Controlled-PDF scheduler applied a speciﬁc case ofmulti-level scheduling for two-level cache hierarchies with private caches and a single shared cache, where the
PDF scheduling was applied at each level.
For machines with multiple shared caches, such as multi-socket, multi-core architectures, Chen et al. [130]proposed CATS, which introduced triple-level work stealing. Triple-level work stealing is also a speciﬁccase of multi-level scheduling for multi-socket, multi-core architectures, where conventional work stealing is121

8. Related Workapplied at each level. In CATS, the working set sizes for each task are estimated by online proﬁling for iterative
computations. LAWS [128], [129] is an improved version of CATS for NUMA architectures. It incorporates
an initial task mapping at the level of NUMA nodes to promote local memory accesses. The initial task
mapping of LAWS is diﬀerent from that of ADWS in that the deterministic task mapping in ADWS can beapplied to every level of a cache in a more general, uniﬁed way, whereas LAWS handles only NUMA nodes.
All of the above approaches assume speciﬁc hardware conﬁgurations with a ﬁxed number of cache levels.
The space-bounded (SB) scheduler [117], [125], [126] was proposed for arbitrary cache hierarchies, andtheoretical analysis was provided. Simhadri et al. [133], [134] conducted the ﬁrst experimental analysis of the
SB scheduler and showed that it could reduce shared cache misses in practice. The idea of the SB scheduler
is similar to that of multi-level scheduling in that it ties a task to a cache so that the task’s descendants are
executed only within that cache. On the other hand, the SB scheduler diﬀers from multi-level schedulingin that it can tie multiple tasks to a cache as long as the tasks’ total working set size does not exceed thecache capacity. Accordingly, as indicated by Blelloch et al. [117], the SB scheduler is expected to be morerobust than multi-level scheduling against irregular computations, and our experimental results conﬁrmed thatﬁnding. However, our evaluation also revealed that the SB scheduler caused more cache misses than multi-level
scheduling in practice. A variant of the SB scheduler for programs with dynamic memory allocation has also
been studied [189]. Nevertheless, none of the SB schedulers have addressed iterative data locality.
On the other hand, the issue of iterative data locality was pointed out by Acar et al. [27], and they proposed
locality-guided work stealing to resolve the issue by setting an aﬃnity for each task. The aﬃnity speciﬁes
the preferred places at which a task should be executed, and many approaches have adopted concepts thatare similar to aﬃnity [172], [190]–[195]. To determine the execution places, some approaches [172], [191],[194] require programmer-speciﬁed hints about places, and Drebes et al. [192], [193] used data dependency
information obtained from a data-ﬂow-based parallel runtime system (OpenStream [196]). In contrast, ADWS
exploits iterative data locality through deterministic scheduling, and the hints for ADWS are oblivious to
hardware-speciﬁc properties such as places.
Constrained work stealing [127] is also based on deterministic scheduling, in which the execution ofnested parallel programs is traced and replayed for iterative programs. It utilizes a steal tree [197], which
is a lightweight tracing mechanism for a work stealing schedule, and it allows for dynamic work stealing in
addition to replay to ﬁx load imbalances. However, because it uses conventional random work stealing toschedule the ﬁrst iteration, it does not consider hierarchical data locality in deep cache hierarchies. In contrast,
ADWS addresses both hierarchical and iterative data locality on arbitrary cache hierarchies in a uniﬁed way.
In addition, many studies have improved steal strategies for work stealing without relying on any schedulinghints by the programmer. Min et al. [58] proposed hierarchical victim selection, which attempts to steal fromthe nearest workers in hierarchical order, and Drebes et al. [192] adopted a similar approach in their system.
Probability Work Stealing (PWS), proposed by Quintin and Wagner [131], changes the probability of choosing
victims for stealing so that close workers are preferably chosen as victims. These heuristic methods cansomewhat improve cache utilization, but the performance improvement is marginal as previously studied [1],[133], [134]. Quintin and Wagner [131] also proposed Hierarchical Work Stealing (HWS), which is similar to
the multi-level scheduling applied in this dissertation but operates on distributed memory. It splits tasks into
global tasks, which can be stolen across nodes, and local tasks, which can only be stolen within a node. Thisidea of splitting tasks into two levels on distributed memory was also studied in [78], [198]. HWS was designed
to reduce stealing across nodes rather than to limit the working set sizes for shared caches; accordingly, notmuch improvement in cache utilization is expected, because it determines global tasks on the basis of theirdepths rather than the working set sizes.

1229. Conclusions
This dissertation has demonstrated that fork-join task parallelism could be eﬃciently scaled to multiple
nodes. Notably, the Itoyori implementation of ExaFMM, a real-world application for N-body simulation,
showed up to a 10× speedup for strong scaling from a single node to 12 nodes, and maintained 75% paralleleﬃciency on 512 nodes (24,576 cores) for weak scaling. Such high performance of Itoyori was made possible
mainly by the following three factors: eﬃcient distributed multithreading and work-stealing implementations(Chapter 4), software caching for global memory access (Chapter 5), and locality-aware task scheduling by
ADWS (Chapter 6). In the following, we brieﬂy review each factor and our contributions.
In this dissertation, we ﬁrst investigated distributed multithreading for task-parallel execution, which is afundamental component in Itoyori. Itoyori’s thread implementation is based on the uni-address scheme [28],[29]. We provided several improvements to it, including an eﬃcient RDMA-based join implementation. An
important ﬁnding in our evaluation is that ﬂexible multithreading (supporting dynamic thread migration atfork/join calls) can be realized on distributed memory without sacriﬁcing performance. Notably, it demonstrated 96.4% parallel eﬃciency on 110,594 cores in the UTS benchmark. In addition, the ﬂexibility inthreading operations (e.g., continuation stealing and dynamic suspension) had overall performance beneﬁts
in programs with complicated task dependencies, compared with inﬂexible existing task-parallel implementations. Furthermore, the capability of dynamic thread migration helped the implementation of ADWS on
distributed memory.
We then challenged the data locality issue in a naive combination of the PGAS and fork-join models. The ﬁrst
step to reconciliation between them was to introduce a software cache to the PGAS system. The conﬂict wasthat tasks could be dynamically migrated in fork-join parallelism, while conventional PGAS systems required
programmers explicitly manage data movement. Consequently, the remote data shared by multiple tasks could
not be easily kept in the local memory and reused among multiple tasks. By incorporating a software cache,eﬃcient data reuse was enabled across multiple tasks that are scheduled on the same process. Furthermore,
we have introduced checkout/checkin APIs for space-eﬃcient access to the cached data. As a result, softwarecaching improved performance by up to 9.8× in ExaFMM.
The second step to reconciliation was to employ a locality-aware task scheduler, ADWS, for fork-joinparallelism. The conﬂict was that fork-join parallelism relied on nondeterministic scheduling based on
randomness, while the PGAS model required programmers compute on local memory as much as possible.
ADWS resolved this conﬂict by performing deterministic task mapping, so that the task placement matches thememory placement. At the same time, it can dynamically load-balance tasks by localized work stealing; thisis why ADWS stands for almost deterministic work stealing. As a result, ADWS improved performance by
up to 2.2× in Cilksort and enabled the Heat benchmark to achieve over 95% parallel eﬃciency on 576 nodes(27,648 cores) for weak scaling.
In addition to high performance, Itoyori demonstrated great productivity in developing distributed-memory
applications. As explained in Chapter 3, Itoyori enables high-level programming over distributed memory,
without explicit communication or distinction between shared and distributed memory. This programming
model is much more like a shared-memory model, except for global memory accesses via checkout/checkin1239. Conclusions
APIs. Evidently, the applications used in our evaluation were all straightforwardly ported from existing
shared-memory fork-join programs to Itoyori. The porting process did not involve any change for the logicof their high-level parallel algorithms, which would be required if we used diﬀerent paradigms such as MPI.
Furthermore, our performance evaluation revealed that Itoyori performed competitively to the corresponding
MPI+OpenMP implementations overall. Therefore, Itoyori is expected to achieve both high productivity andperformance in distributed-memory computing. Taking these results into account, we can say that Itoyori hasachieved a high level of both productivity and performance in distributed memory environments.
Nevertheless, we consider that there is still much room for improvements in the Itoyori runtime system, aslisted in the following:
Sharing a software cache by multiple processes. Currently, only private caches are implemented in Itoy-ori, without sharing a software cache among multiple processes within the same node. In fact, checkout/checkin APIs were designed with a shared cache in mind; the same shared cache block can bemapped to multiple global views in diﬀerent processes, further reducing space usage compared with
GET/PUT APIs. Nevertheless, implementing general shared caches would be challenging, as it would
require reconsideration for the cache coherence protocol. Thus, the ﬁrst step would be implementing ashared cache only for read-only data.
Implementing multi-level scheduling. Itoyori currently implements only single-level ADWS. Supporting

multi-level scheduling would greatly improve the cache utilization, but this will become beneﬁcial only
after implementing a shared software cache.
More eﬃcient cache coherence protocols. Currently, all caches are conservatively self-invalidated at ac-quire fences. Therefore, even for read-only data, caches are invalidated every time work stealing events
happen. Better cache coherence protocols would reduce the number of cache misses.
Context switching upon cache misses. When a cache miss happens, the local process is blocked until

communication is completed in the current implementation. Instead, the process can context switch toanother task while hiding the communication latency.
Adaptive sub-block sizes for fetching remote data. The best sub-block size for fetching remote data

varies across applications and even subroutines. If the system can automatically tune the sub-block
sizes by online proﬁling or some sort of it, it would greatly reduce the burden on users to manuallyconﬁgure sub-block sizes (and possibly other parameters).
More user-friendly programming interfaces. As discussed in Section 3.5, the current Itoyori programming

model has some nontrivial pitfalls. Designing a new language or developing tools for compile-time or
runtime veriﬁcation will greatly improve the productivity of application developers in the future.
Proﬁlers to identify the bottleneck. The current Itoyori implementation only has a simple proﬁler thatshows the performance breakdown as shown in Chapter 7. Given that there are proﬁlers like Cilkprof [199]and DAGViz [200] for fork-join parallelism on shared memory, it is worth considering the developmentof similar tools tailored for Itoyori.

124

Appendix A.
Itoyori Program Examples
A.1. Cilksort
Figure A.1 shows the Cilksort program rewritten in C++ with Itoyori APIs. It uses the ityr::global spancontainer to represent a contiguous global memory region as a pair of its global address and size. The input
spans are recursively divided into smaller spans according to the divide-and-conquer strategy. The cilksort()
function takes two spans a and b and sorts the elements in a by using b as a temporary buﬀer. First, it logicallysplits both a and b into four equal-sized spans (line 7–12), each of which is then sorted recursively in parallel(line 13–17). The parallel invoke() function forks multiple closures (lambda expressions) as parallel tasks
and returns control when all these tasks are joined. After sorting for the four spans is completed, two pairs ofthem are merged into the temporary buﬀers in parallel (line 18–20). Then, they are merged into the original
span a (line 21). The recursion continues until the span becomes suﬃciently small (less than the cutoff value).
At the leaf task, it switches to the serial std::sort() (line 5) by checking out the array in the read-write
mode. The checkout span returned at line 21 is destroyed after std::sort() is completed, automaticallycalling the checkin API. Similarly, the arrays for the merge operation are checked out with proper checkout
modes (line 27–30).
Figure A.2 shows the binary search function used in the Cilksort code. It takes as an argument a global
reference for the target value, which is fetched by using the GET API (line 5). For small values, using GET/PUT
APIs is often more eﬃcient than using checkout/checkin APIs, because checkout/checkin APIs involve two
API calls even for read-only or write-only access, while GET/PUT APIs require only one. However, GET/PUT
APIs can be used only for trivially copyable types in C++. During binary search, values within the input array
are repeatedly fetched through the GET API (line 10).

125Appendix A. Itoyori Program Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

template <typename T>
void cilksort(ityr::global_span<T> a, ityr::global_span<T> b) {if (a.size() < cutoff) {auto a_cs = ityr::make_checkout(a, ityr::checkout_mode::read_write);
std::sort(a_cs.begin(), a_cs.end());} else {auto [a12, a34] = split_two(a);auto [a1 , a2 ] = split_two(a12);
auto [a3 , a4 ] = split_two(a34);
auto [b12, b34] = split_two(b);auto [b1 , b2 ] = split_two(b12);
auto [b3 , b4 ] = split_two(b34);
ityr::parallel_invoke([=] { cilksort(a1, b1); },// sort a1[=] { cilksort(a2, b2); },// sort a2[=] { cilksort(a3, b3); },// sort a3[=] { cilksort(a4, b4); });// sort a4
ityr::parallel_invoke([=] { cilkmerge(a1, a2, b12); }, // merge a1 and a2 -> b12[=] { cilkmerge(a3, a4, b34); }); // merge a3 and a4 -> b34
cilkmerge(b12, b34, a);// merge b12 and b34 -> a}}template <typename T>
void cilkmerge(ityr::global_span<T> s1, ityr::global_span<T> s2, ityr::global_span<T> d) {if (d.size() < cutoff) {auto [s1_cs, s2_cs, d_cs] =ityr::make_checkouts(s1, ityr::checkout_mode::read,s2, ityr::checkout_mode::read,d , ityr::checkout_mode::write);
std::merge(s1_cs.begin(), s1_cs.end(), s2_cs.begin(), s2_cs.end(), dest_cs.begin());} else {size_t p1 = (s1.size() + 1) / 2;
size_t p2 = binary_search(s2, s1[p1 - 1]);
auto [s11, s12] = split_at(s1, p1);
auto [s21, s22] = split_at(s2, p2);
auto [d1 , d2 ] = split_at(d , p1 + p2);
ityr::parallel_invoke([=] { cilkmerge(s11, s21, d1); }, // merge s11 and s21 -> d1[=] { cilkmerge(s12, s22, d2); }); // merge s12 and s22 -> d2}}
int main() {size_t n = /* The number of array elements */;ityr::init();ityr::root_exec([=] {// Initialize global vectors and spans
ityr::global_vector<int> a_vec(n), b_vec(n);ityr::global_span<int> a(a_vec), b(b_vec);
init_array_with_random_values(a);// Sort the arraycilksort(a, b);// Check if the array is correctly sortedbool success = ityr::is_sorted(ityr::execution::par, a.begin(), a.end());
std::cout << (success ? "OK" : "Fail") << std::endl;});
ityr::fini();}Figure A.1.: Cilksort program written with Itoyori in C++.
126

Appendix A. Itoyori Program Examples

1
2
3
4
5
6
7
8
9
10
11
12
13
14

template <typename T>
size_t binary_search(ityr::global_span<T> a, ityr::ori::global_ref<T> r) {// The GET operation can be used only for trivially copyable types
static_assert(std::is_trivially_copyable_v<T>);
T v = r.get();size_t l = 0;
size_t h = a.size();
while (l < h) {size_t m = l + (h - l) / 2;
if (v <= a[m].get()) h = m;else
l = m + 1;}
return h;}

Figure A.2.: Binary search program with Itoyori in C++.

127

Appendix B.
Supplemental Experimental Results
This appendix chapter includes supplemental experimental results that have not been covered in Chapter 7.

B.1. ANNS Graph Construction with the msspacev-10M Dataset
In Section 7.6, we have discussed the performance of ANNS graph construction with the bigann-10M dataset.
We also conducted experiments with the msspacev-10M dataset (included in the big-ann-benchmarks dataset
repository [160] as well). Although we excluded msspacev-10M from the main text because it showed similar
trends to bigann-10M, we put the results in this appendix section. Figure B.1, Figure B.2, Figure B.3,
and Figure B.4 show execution time, parallel eﬃciency, proﬁling results, and QPS-recall curves for the
msspacev-10M dataset. They correspond to Figure 7.22, Figure 7.23, Figure 7.24, and Figure 7.25 (results forthe bigann-10M dataset) in Section 7.6.

B.2. Proﬁling Result of PageRank
Figure B.5 shows the proﬁling result for the rmat26 dataset with strong scaling. In Section 7.7, we have only
shown the proﬁling result for the twitter2010 dataset because both results are similar.

128

Appendix B. Supplemental Experimental Results

1000

Execution time (s)Execution time (s)

1000

100

100

10

10
1

10

1

# of nodes (× 76 cores/node)(b) SQUID-C, HCNNG.

Execution time (s)Execution time (s)(a) SQUID-C, HNSW.

1000

100

1

10

# of nodes (× 76 cores/node)Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

1000

100

10

1

# of nodes (× 48 cores/node)

10# of nodes (× 48 cores/node)(c) Wisteria-O, HNSW.

(d) Wisteria-O, HCNNG.

Figure B.1.: Execution time of ANNS graph construction with strong scaling (msspacev-10M). The straightdashed lines represent the ideal speedup (calculated based on the serial execution time).
100

Parallel Efficiency (%)

Parallel Efficiency (%)

100
80
60
40
20
0

80
60
40
20
0

1

10

1

# of nodes (× 76 cores/node)(a) SQUID-C, HNSW.

(b) SQUID-C, HCNNG.
100

Parallel Efficiency (%)

100

Parallel Efficiency (%)

10# of nodes (× 76 cores/node)

80
60
40
20
0

Random WS, GET/PUT (no-cache)Random WS, Checkout/Checkin
ADWS, GET/PUT (no-cache)
ADWS, Checkout/Checkin

80
60
40
20
0

1

10

# of nodes (× 48 cores/node)(c) Wisteria-O, HNSW.

1

10# of nodes (× 48 cores/node)(d) Wisteria-O, HCNNG.

Figure B.2.: Parallel eﬃciency of ANNS graph construction with strong scaling (msspacev-10M). Paralleleﬃciency of the executions in Figure 7.16 is calculated based on the best one-node execution time
among the execution policies.
129

Time ratio

Appendix B. Supplemental Experimental Results

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, HNSW.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, HCNNG.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program36# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, HNSW.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program36# of nodes (× 48 cores/node)(d) Wisteria-O, HCNNG.

Figure B.3.: Proﬁling of ANNS graph construction with strong scaling (msspacev-10M).

130

Query Per Second (QPS)

Query Per Second (QPS)

Appendix B. Supplemental Experimental Results

1M

100k

HCNNG (Itoyori)
HCNNG (PBBS)HNSW (Itoyori)
HNSW (PBBS)

10k
0.0

0.2

0.4

0.6

Recall 10@10

0.8

1.0

100k

HNSW (Itoyori)
HNSW (PBBS)

1000
0.0

(a) SQUID-C (76 cores).

HCNNG (Itoyori)
HCNNG (PBBS)

10k

0.2

0.4

0.6

Recall 10@10

0.8

1.0

(b) Wisteria-O (48 cores).

Figure B.4.: Query per second (QPS) on one node with various recall rates (msspacev-10M). Shown to checkthe validity of the graph obtained as a result.

131

Time ratio

Appendix B. Supplemental Experimental Results

Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(a) SQUID-C, Itoyori-Naive.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

4

8

16

0

1

2

4

8

16

0

ADWS, GET/PUT (no-cache)

1

2

4

8

16

1

0

ADWS, Checkout/Checkin

1

2

4

8

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program16# of nodes (× 76 cores/node)

Time ratio(b) SQUID-C, Itoyori-GPOP.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser Program36# of nodes (× 48 cores/node)

Time ratio(c) Wisteria-O, Itoyori-Naive.
Random WS, GET/PUT (no-cache) Random WS, Checkout/Checkin
1
1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0

1

2

6

12

36

0

1

2

6

12

36

0

ADWS, GET/PUT (no-cache)

1

2

6

12

36

1

0

ADWS, Checkout/Checkin

1

2

6

12

36

# of nodes (× 48 cores/node)(d) Wisteria-O, Itoyori-GPOP.

Figure B.5.: Proﬁling of PageRank (3 iterations) with strong scaling (rmat26).

132

Sched Overheads
Sched Loop
GET
PUT
Checkout
Checkin
ReleaseAcquireUser ProgramList of Publications[1] S. Shiina and K. Taura, “Almost deterministic work stealing,” in Proceedings of the International
Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19, Denver,
Colorado, USA, 2019, 47:1–47:16.
[2] S. Shiina, S. Iwasaki, K. Taura, and P. Balaji, “Lightweight preemptive user-level threads,” in Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming,
PPoPP ’21, Virtual Event, Republic of Korea, 2021, pp. 374–388.
[3] S. Shiina and K. Taura, “Improving cache utilization of nested parallel programs by almost deterministic
work stealing,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 12, pp. 4530–4546,
2022.
[4] S. Shiina and K. Taura, “Distributed continuation stealing is more scalable than you might think,” inProceedings of the 2022 IEEE International Conference on Cluster Computing, Cluster ’22, Heidelberg, Germany, 2022, pp. 129–141.
[5] S. Shiina and K. Taura, “Itoyori: Reconciling global address space and global fork-join task parallelism,”in Proceedings of the International Conference for High Performance Computing, Networking, Storage
and Analysis, SC ’23, Denver, Colorado, USA, 2023, 14:1–14:15.

133

References[6] MPI: A message-passing interface standard version 4.1, Message Passing Interface Forum, 2023.
[Online]. Available: https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf.
[7] OpenMP application program interface version 5.2, OpenMP Architecture Review Board, 2021. [Online]. Available: https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification5-2.pdf.
[8] R. D. Blumofe, C. F. Joerg, B. C. Kuszmaul, C. E. Leiserson, K. H. Randall, and Y. Zhou, “Cilk: An
eﬃcient multithreaded runtime system,” Journal of Parallel and Distributed Computing, vol. 37, no. 1,
pp. 55–69, 1996.
[9] M. Frigo, C. E. Leiserson, and K. H. Randall, “The implementation of the Cilk-5 multithreaded
language,” in Proceedings of the ACM SIGPLAN Conference on Programming Language Design and
Implementation, PLDI ’98, Montreal, Quebec, Canada, 1998, pp. 212–223.
[10]

T. B. Schardl and I.-T. A. Lee, “OpenCilk: A modular and extensible software infrastructure for fasttask-parallel code,” in Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles andPractice of Parallel Programming, PPoPP ’23, Montreal, QC, Canada, 2023, pp. 189–203.

[11]

oneAPI threading building blocks (oneTBB), Intel Corporation, 2023. [Online]. Available: https://oneapi-src.github.io/oneTBB/.

[12]

J. Reinders, Intel Threading Building Blocks: Outﬁtting C++ for Multi-Core Processor Parallelism.
O’Reilly Media, 2007.

[13]

Parallel patterns library (PPL), Microsoft, 2021. [Online]. Available: https://learn.microsoft.
com/en-us/cpp/parallel/concrt/parallel-patterns-library-ppl.

[14]

T. Hiraishi, M. Yasugi, S. Umatani, and T. Yuasa, “Backtracking-based load balancing,” in Proceedingsof the 14th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP’09, Raleigh, North Carolina, USA, 2009, pp. 55–64.

[15]

J. Nakashima and K. Taura, “MassiveThreads: A thread library for high productivity languages,”
Concurrent Objects and Beyond, vol. 8665, pp. 222–238, 2014.

[16]

B. L. Chamberlain, D. Callahan, and H. P. Zima, “Parallel programmability and the Chapel language,”
International Journal of High Performance Computing Applications, vol. 21, no. 3, pp. 291–312, 2007.

[17]

P. Charles, C. Grothoﬀ, V. Saraswat, et al., “X10: An object-oriented approach to non-uniform clustercomputing,” in Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, San Diego, California, USA, 2005,
pp. 519–538.

[18]

D. Lea, “A Java fork/join framework,” in Proceedings of the ACM 2000 conference on Java Grande,
JAVA ’00, San Francisco, California, USA, 2000, pp. 36–43.

134

References[19]

A. A. A. Donovan and B. W. Kernighan, The Go programming language. Addison-Wesley Professional,
2015.

[20]

J. Bezanson, J. Nash, and K. Pamnany, Announcing composable multi-threaded parallelism in Julia,
2019. [Online]. Available: https://julialang.org/blog/2019/07/multithreading/.

[21]

X. Leroy, D. Doligez, A. Frisch, et al., The OCaml system release 5.1 – documentation and user’s
manual, 2023. [Online]. Available: https : / / v2 . ocaml . org / releases / 5 . 1 / ocaml - 5 . 1 refman.pdf.

[22]

Task parallel library (TPL), Microsoft, 2022. [Online]. Available: https://learn.microsoft.com/
en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl.

[23]

A. Ohori, K. Taura, and K. Ueno, Making SML# a general-purpose high-performance language, ML
Family Workshop 2017, 2017. [Online]. Available: https://www.pllab.riec.tohoku.ac.jp/
papers/ml2017Preversion.pdf.

[24]

R. Raghunathan, S. K. Muller, U. A. Acar, and G. Blelloch, “Hierarchical memory management forparallel programs,” in Proceedings of the 21st ACM SIGPLAN International Conference on Functional
Programming, ICFP ’16, Nara, Japan, 2016, pp. 392–406.

[25]

J. Arora, S. Westrick, and U. A. Acar, “Provably space-eﬃcient parallel functional programming,”
Proceedings of the ACM on Programming Languages, vol. 5, no. POPL, 18:1–18:33, 2021.

[26]

R. D. Blumofe and C. E. Leiserson, “Scheduling multithreaded computations by work stealing,”
Journal of the ACM, vol. 46, no. 5, pp. 720–748, 1999.

[27]

U. A. Acar, G. E. Blelloch, and R. D. Blumofe, “The data locality of work stealing,” in Proceedings of
the Twelfth Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA ’00, Bar Harbor,
Maine, USA, 2000, pp. 1–12.

[28]

S. Akiyama and K. Taura, “Uni-address threads: Scalable thread management for RDMA-based work
stealing,” in Proceedings of the 24th International Symposium on High-Performance Parallel andDistributed Computing, HPDC ’15, Portland, Oregon, USA, 2015, pp. 15–26.

[29]

S. Akiyama and K. Taura, “Scalable work stealing of native threads on an x86-64 inﬁniband cluster,”
Journal of Information Processing, vol. 24, no. 3, pp. 583–596, 2016.

[30]

V. Nagarajan, D. J. Sorin, M. D. Hill, and D. A. Wood, A Primer on Memory Consistency and Cache
Coherence – Second Edition. Morgan & Claypool Publishers, 2020.

[31]

M. Frigo, C. E. Leiserson, H. Prokop, and S. Ramachandran, “Cache-oblivious algorithms,” in Proceedings of the 40th Annual Symposium on Foundations of Computer Science, FOCS ’99, New York,New York, USA, 1999, pp. 285–297.

[32]

M. Frigo, C. E. Leiserson, H. Prokop, and S. Ramachandran, “Cache-oblivious algorithms,” ACM
Transactions on Algorithms, vol. 8, no. 1, 4:1–4:22, 2012.

[33]

M. Gorman, Foundation for automatic NUMA balancing, 2012. [Online]. Available: https://lwn.
net/Articles/523065/.

[34]

InﬁniBand architecture speciﬁcation, InﬁniBand Trade Association. [Online]. Available: https://www.infinibandta.org/ibta-specification/.

[35]

Y. Ajima, T. Kawashima, T. Okamoto, et al., “The Tofu interconnect D,” in Proceedings of the 2018
IEEE International Conference on Cluster Computing, Cluster ’18, Belfast, UK, 2018, pp. 646–654.

135

References[36]

S. Di Girolamo, F. Vella, and T. Hoeﬂer, “Transparent caching for RMA systems,” in Proceedingsof the 31st IEEE International Parallel and Distributed Processing Symposium, IPDPS ’17, Orlando,
Florida, USA, 2017, pp. 1018–1027.

[37]

S. Kaxiras, D. Klaftenegger, M. Norgren, A. Ros, and K. Sagonas, “Turning centralized coherence
and distributed critical-section execution on their head: A new approach for scalable distributed sharedmemory,” in Proceedings of the 24th International Symposium on High-Performance Parallel andDistributed Computing, HPDC ’15, Portland, Oregon, USA, 2015, pp. 3–14.

[38]

MPICH: High-performance portable MPI, Argonne National Laboratory, 2023. [Online]. Available:
https://www.mpich.org.

[39]

Open MPI: Open source high performance computing, The Open MPI Project, 2023. [Online]. Available: https://www.open-mpi.org.

[40]

T. Hoeﬂer, J. Dinan, R. Thakur, et al., “Remote memory access programming in MPI-3,” ACM
Transactions on Parallel Computing, vol. 2, no. 2, pp. 1–26, 2015.

[41]

J. Schuchart, A. Bouteiller, and G. Bosilca, “Using MPI-3 RMA for active messages,” in Proceedingsof the 2019 IEEE/ACM Workshop on Exascale MPI, ExaMPI ’19, Denver, Colorado, USA, 2019,
pp. 47–56.

[42]

E. Ayguadé, N. Copty, A. Duran, et al., “The design of OpenMP tasks,” IEEE Transactions on Paralleland Distributed Systems, vol. 20, no. 3, pp. 404–418, 2008.

[43]

Y. Guo, R. Barik, R. Raman, and V. Sarkar, “Work-ﬁrst and help-ﬁrst scheduling policies for asyncﬁnish task parallelism,” in Proceedings of the 23rd IEEE International Symposium on Parallel andDistributed Processing Symposium, IPDPS ’09, Rome, Italy, 2009.

[44]

R. P. Brent, “The parallel evaluation of general arithmetic expressions,” Journal of the ACM, vol. 21,
no. 2, pp. 201–206, 1974.

[45]

K. B. Wheeler, R. C. Murphy, and D. Thain, “Qthreads: An API for programming with millionsof lightweight threads,” in Proceedings of the 22nd IEEE International Parallel and Distributed
Processing Symposium, IPDPS ’08, Miami, Florida, USA, 2008, pp. 1–8.

[46]

S. Seo, A. Amer, P. Balaji, et al., “Argobots: A lightweight low-level threading and tasking framework,”
IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 3, pp. 512–526, 2017.

[47]

W. Endo, S. S. Sato, and K. Taura, “ComposableThreads: Rethinking user-level threads with composability and parametricity in C++,” Journal of Information Processing, vol. 30, pp. 269–282, 2022.

[48]

S. Iwasaki, A. Amer, K. Taura, and P. Balaji, “Lessons learned from analyzing dynamic promotion foruser-level threading,” in Proceedings of the International Conference for High Performance Computing,
Networking, Storage, and Analysis, SC ’18, Dallas, Texas, USA, 2018, 23:1–23:12.

[49]

S. Iwasaki, A. Amer, K. Taura, and P. Balaji, “Analyzing the performance trade-oﬀ in implementing
user-level threads,” IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 8, pp. 1859–
1877, 2020.

[50]

S. Chen, P. B. Gibbons, M. Kozuch, et al., “Scheduling threads for constructive cache sharing on CMPs,”in Proceedings of the Nineteenth Annual ACM Symposium on Parallel Algorithms and Architectures,
SPAA ’07, San Diego, California, USA, 2007, pp. 105–115.

136

References[51]

G. E. Blelloch, P. B. Gibbons, and Y. Matias, “Provably eﬃcient scheduling for languages with ﬁnegrained parallelism,” Journal of the ACM, vol. 46, no. 2, pp. 281–321, 1999.

[52]

V. Kumar, Y. Zheng, V. Cavé, Z. Budimlić, and V. Sarkar, “HabaneroUPC++: A compiler-free PGAS
library,” in Proceedings of the 8th International Conference on Partitioned Global Address Space
Programming Models, PGAS ’14, Eugene, Oregon, USA, 2014, 5:1–5:10.

[53]

M. Grossman, V. Kumar, Z. Budimlić, and V. Sarkar, “Integrating asynchronous task parallelism with
OpenSHMEM,” in Proceedings of the Third Workshop on OpenSHMEM and Related Technologies,
OpenSHMEM ’16, Baltimore, Maryland, USA, 2016, pp. 3–17.

[54]

J. Dinan, S. Krishnamoorthy, D. B. Larkins, J. Nieplocha, and P. Sadayappan, “Scioto: A frameworkfor global-view task parallelism,” in Proceedings of the 37th International Conference on Parallel
Processing, ICPP ’08, Portland, Oregon, USA, 2008, pp. 586–593.

[55]

J. Dinan, D. B. Larkins, P. Sadayappan, S. Krishnamoorthy, and J. Nieplocha, “Scalable work stealing,”
in Proceedings of the International Conference for High Performance Computing, Networking, Storage,
and Analysis, SC ’09, Portland, Oregon, USA, 2009, 53:1–53:11.

[56]

H. Cartier, J. Dinan, and D. B. Larkins, “Optimizing work stealing communication with structured
atomic operations,” in Proceedings of the 50th International Conference on Parallel Processing, ICPP
’21, Lemont, Illinois, USA, 2021, 36:1–36:10.

[57]

T. Gautier, X. Besseron, and L. Pigeon, “KAAPI: A thread scheduling runtime system for data ﬂow
computations on cluster of multi-processors,” in Proceedings of the 2007 International Workshop on
Parallel Symbolic Computation, PASCO ’07, London, Ontario, Canada, 2007, pp. 15–23.

[58]

S.-J. Min, C. Iancu, and K. Yelick, “Hierarchical work stealing on manycore clusters,” in Proceedings ofthe Fifth Conference on Partitioned Global Address Space Programming Models, PGAS ’11, Galveston
Island, Texas, USA, 2011, pp. 1–10.

[59]

J. Nelson, B. Holt, B. Myers, et al., “Latency-tolerant software distributed shared memory,” in Proceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC ’15, Denver, Colorado,
USA, 2015, pp. 291–305.

[60]

K. H. Randall, “Cilk: Eﬃcient multithreaded computing,” Ph.D. dissertation, Department of ElectricalEngineering and Computer Science, Massachusetts Institute of Technology, 1998.

[61]

P. Keleher, A. L. Cox, S. Dwarkadas, and W. Zwaenepoel, “TreadMarks: Distributed shared memory
on standard workstations and operating systems,” in Proceedings of the USENIX Winter 1994 TechnicalConference on USENIX Winter 1994 Technical Conference, WTEC ’94, San Francisco, California,
USA, 1994.

[62]

P. Keleher, A. L. Cox, and W. Zwaenepoel, “Lazy release consistency for software distributed sharedmemory,” in Proceedings of the 19th Annual International Symposium on Computer Architecture,
ISCA ’92, Queensland, Australia, 1992, pp. 13–21.

[63]

B. Bershad, M. Zekauskas, and W. Sawdon, “The Midway distributed shared memory system,” inDigest of Papers. The 38th IEEE Computer Society International Conference, COMPCON Spring ’93,
San Francisco, California, USA, 1993, pp. 528–537.

[64]

W. Endo, S. Sato, and K. Taura, “MENPS: A decentralized distributed shared memory exploiting
RDMA,” in Proceedings of 2020 IEEE/ACM Fourth Annual Workshop on Emerging Parallel andDistributed Runtime Systems and Middleware, IPDRM ’20, Virtual Event, 2020, pp. 9–16.

137

References[65]

S. V. Adve and M. D. Hill, “Weak ordering – a new deﬁnition,” in Proceedings of the 17th Annual
International Symposium on Computer Architecture, ISCA ’90, Seattle, Washington, USA, 1990,
pp. 2–14.

[66]

L. Lamport, “How to make a multiprocessor computer that correctly executes multiprocess program,”
IEEE Computer Architecture Letters, vol. 28, no. 09, pp. 690–691, 1979.

[67]

R. D. Blumofe, M. Frigo, C. F. Joerg, C. E. Leiserson, and K. H. Randall, “Dag-consistent distributed
shared memory,” in Proceedings of the 10th International Parallel Processing Symposium, IPPS ’96,
Honolulu, Hawaii, USA, 1996, pp. 132–141.

[68]

R. D. Blumofe, M. Frigo, C. F. Joerg, C. E. Leiserson, and K. H. Randall, “An analysis of dag-consistent
distributed shared-memory algorithms,” in Proceedings of the Eighth Annual ACM Symposium on
Parallel Algorithms and Architectures, SPAA ’96, Padua, Italy, 1996, pp. 297–308.

[69]

R. W. Numrich and J. Reid, “Co-array fortran for parallel programming,” SIGPLAN Fortran Forum,
vol. 17, no. 2, pp. 1–31, 1998.

[70]

T. El-Ghazawi, W. Carlson, T. Sterling, and K. Yelick, UPC: Distributed Shared Memory Programming.
John Wiley & Sons, 2005.

[71]

J. Lee and M. Sato, “Implementation and performance evaluation of XcalableMP: A parallel programming language for distributed memory systems,” in Proceedings of the 39th International Conference
on Parallel Processing Workshops, ICPPW ’10, San Diego, California, USA, 2010, pp. 413–420.

[72]

B. Chapman, T. Curtis, S. Pophale, et al., “Introducing OpenSHMEM: SHMEM for the PGAS community,” in Proceedings of the Fourth Conference on Partitioned Global Address Space Programming
Model, PGAS ’10, New York, New York, USA, 2010, pp. 1–3.

[73]

J. Nieplocha, R. J. Harrison, and R. J. Littleﬁeld, “Global Arrays: A nonuniform memory accessprogramming model for high-performance computers,” The Journal of Supercomputing, vol. 10, no. 2,
pp. 169–189, 1996.

[74]

Y. Zheng, A. Kamil, M. B. Driscoll, H. Shan, and K. Yelick, “UPC++: A PGAS extension for C++,” inProceedings of the 28th IEEE International Parallel and Distributed Processing Symposium, IPDPS’14, Phoenix, Arizona, USA, 2014, pp. 1105–1114.

[75]

K. Fuerlinger, T. Fuchs, and R. Kowalewski, “DASH: A C++ PGAS library for distributed data
structures and parallel algorithms,” in Proceedings of the 2016 IEEE 18th International Conference
on High Performance Computing and Communications, HPCC ’16, Sydney, NSW, Australia, 2016,
pp. 983–990.

[76]

H. Kaiser, T. Heller, B. Adelstein-Lelbach, A. Serio, and D. Fey, “HPX: A task based programming
model in a global address space,” in Proceedings of the 8th International Conference on Partitioned
Global Address Space Programming Models, PGAS ’14, Eugene, Oregon, USA, 2014, 6:1–6:11.

[77]

S. Hiranandani, K. Kennedy, and C.-W. Tseng, “Compiling fortran D for MIMD distributed-memorymachines,” Communications of the ACM, vol. 35, no. 8, pp. 66–80, 1992.

[78]

J. Paudel, O. Tardieu, and J. N. Amaral, “On the merits of distributed work-stealing on selective
locality-aware tasks,” in Proceedings of the 42nd International Conference on Parallel Processing,
ICPP ’13, Lyon, France, 2013, pp. 100–109.

138

References[79]

W. Zhang, O. Tardieu, D. Grove, et al., “Glb: Lifeline-based global load balancing library in X10,” inProceedings of the First Workshop on Parallel Programming for Analytics Applications, PPAA ’14,
Orlando, Florida, USA, 2014, pp. 31–40.

[80]

K. Gharachorloo, D. Lenoski, J. Laudon, P. Gibbons, A. Gupta, and J. Hennessy, “Memory consistency
and event ordering in scalable shared-memory multiprocessors,” in Proceedings of the 17th Annual
International Symposium on Computer Architecture, ISCA ’90, Seattle, Washington, USA, 1990,
pp. 15–26.

[81]

J. Mellor-Crummey, “On-the-ﬂy detection of data races for programs with nested fork-join parallelism,”in Proceedings of the 1991 ACM/IEEE Conference on Supercomputing, SC ’91, Albuquerque, New
Mexico, USA, 1991, pp. 24–33.

[82]

A. Duran, E. Ayguadé, R. M. Badia, et al., “OmpSs: A proposal for programming heterogeneous
multi-core architectures,” Parallel Processing Letters, vol. 21, no. 02, pp. 173–193, 2011.

[83]

V. A. Saraswat, P. Kambadur, S. Kodali, D. Grove, and S. Krishnamoorthy, “Lifeline-based global
load balancing,” in Proceedings of the 16th ACM SIGPLAN Symposium on Principles and Practice ofParallel Programming, PPoPP ’11, San Antonio, Texas, USA, 2011, pp. 201–212.

[84]

S. Olivier, J. Huan, J. Liu, et al., “UTS: An unbalanced tree search benchmark,” in Proceedings of the
19th International Conference on Languages and Compilers for Parallel Computing, LCPC ’06, New
Orleans, Los Angeles, USA, 2006, pp. 235–250.

[85]

L. Kale, B. Acun, S. Bak, et al., The Charm++ parallel programming system, 2019. doi: 10.5281/
zenodo.3370873. [Online]. Available: https://charm.cs.illinois.edu.

[86]

L. V. Kale and G. Zheng, Parallel Science and Engineering Applications: The Charm++ Approach.
CRC Press, Inc., 2013, pp. 1–16.

[87]

O. Tardieu, B. Herta, D. Cunningham, et al., “X10 and APGAS at petascale,” in Proceedings of the
19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’14,
Orlando, Florida, USA, pp. 53–66.

[88]

D. B. Wagner and B. G. Calder, “Leapfrogging: A portable technique for implementing eﬃcient
futures,” in Proceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice ofParallel Programming, PPoPP ’93, San Diego, California, USA, 1993, pp. 208–217.

[89]

K.-F. Faxén, “Eﬃcient work stealing for ﬁne grained parallelism,” in Proceedings of the 39th International Conference on Parallel Processing, ICPP ’10, San Diego, California, USA, 2010, pp. 313–
322.

[90]

E. Mohr, D. A. Kranz, and R. H. Halstead, “Lazy task creation: A technique for increasing the
granularity of parallel programs,” in Proceedings of the 1990 ACM Conference on LISP and Functional
Programming, LFP ’90, Nice, France, 1990, pp. 185–197.

[91]

R. V. Van Nieuwpoort, G. Wrzesińska, C. J. H. Jacobs, and H. E. Bal, “Satin: A high-level and eﬃcientgrid programming model,” ACM Transactions on Programming Languages and Systems, vol. 32, no. 3,
9:1–9:39, 2010.

[92]

G. Antoniu, L. Bougé, and R. Namyst, “An eﬃcient and transparent thread migration scheme in thePM2 runtime system,” in Proceedings of the 11th IPPS/SPDP’99 Workshops Held in Conjunction
with the 13th International Parallel Processing Symposium and 10th Symposium on Parallel andDistributed Processing, RTSPP ’99, San Juan, Puerto Rico, USA, 1999, pp. 496–510.

139

References[93]

C. Huang, O. Lawlor, and L. V. Kalé, “Adaptive MPI,” in Proceedings of the 16th International
Conference on Languages and Compilers for Parallel Computing, LCPC ’03, College Station, Texas,
USA, 2003, pp. 306–322.

[94]

A. Robison, “A primer on scheduling fork-join parallelism with work stealing,” ISO/IEC JTC1/SC22/WG21– The C++ Standards Committee, Tech. Rep. N3872, 2014.

[95]

G. E. Blelloch, P. B. Gibbons, and Y. Matias, “Provably eﬃcient scheduling for languages with ﬁnegrained parallelism,” Journal of the ACM, vol. 46, no. 2, pp. 281–321, 1999.

[96]

F. Schmaus, N. Pfeiﬀer, W. Schröder-Preikschat, T. Hönig, and J. Nolte, “Nowa: A wait-free continuationstealing concurrency platform,” in Proceedings of the 36th IEEE International Parallel and Distributed
Processing Symposium, IPDPS ’21, Lyon, France, 2021.

[97]

G. E. Blelloch and M. Reid-Miller, “Pipelining with futures,” Theory of Computing Systems, vol. 32,
no. 3, pp. 213–239, 1999.

[98]

D. Spoonhower, G. E. Blelloch, P. B. Gibbons, and R. Harper, “Beyond nested parallelism: Tightbounds on work-stealing overheads for parallel futures,” in Proceedings of the Twenty-First Annual
Symposium on Parallelism in Algorithms and Architectures, SPAA ’09, Calgary, Alberta, Canada,2009, pp. 91–100.

[99]

Introduction of ITO, Kyushu University, 2017. [Online]. Available: https : / / www . cc . kyushu u.ac.jp/scp/eng/system/ITO/01 intro.html.

[100] Introduction to the Wisteria/BDEC-01, The University of Tokyo, 2021. [Online]. Available: https://www.cc.u-tokyo.ac.jp/en/supercomputer/wisteria/system.php.
[101] M. Sato, Y. Ishikawa, H. Tomita, et al., “Co-design for A64FX manycore processor and “Fugaku”,” inProceedings of the International Conference for High Performance Computing, Networking, Storage
and Analysis, SC ’20, Atlanta, Georgia, USA, 2020, 47:1–47:15.
[102] P. Shamis, M. G. Venkata, M. G. Lopez, et al., “UCX: An open source framework for HPC network
APIs and beyond,” in Proceedings of the 2015 IEEE 23rd Annual Symposium on High-Performance
Interconnects, HOTI ’15, Santa Clara, California, USA, 2015, pp. 40–43.
[103] C. E. Leiserson, “The Cilk++ concurrency platform,” The Journal of Supercomputing, vol. 51, no. 3,
pp. 244–257, 2010.
[104] G. J. Narlikar, “A parallel, multithreaded decision tree builder,” Carnegie Mellon University, Tech.
Rep. CMU-CS-98-184, 1998.
[105] A. Huynh and K. Taura, “Delay Spotter: A tool for spotting scheduler-caused delays in task parallel
runtime systems,” in Proceedings of the 2017 IEEE International Conference on Cluster Computing,
CLUSTER ’17, Hawaii, USA, 2017, pp. 114–125.
[106] S. Perarnau and M. Sato, “Victim selection and distributed work stealing performance: A case study,”
in Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium, IPDPS’14, Phoenix, Arizona, USA, 2014, pp. 659–668.
[107] H. Parikh, V. Deodhar, A. Gavrilovska, and S. Pande, “Distributed work stealing at scale via matchmaking,” in Proceedings of the 2021 IEEE International Conference on Cluster Computing, Cluster’21, Portland, Oregon, USA, 2021, pp. 250–260.

140

References[108] Y. Sun, G. Zheng, P. Jetley, and L. V. Kalé, “ParSSSE: An adaptive parallel state space search engine,”
Parallel Processing Letters, vol. 21, no. 03, pp. 319–338, 2011.
[109] D. B. Larkins, Saws (GitHub repository), commit hash: cdd36782c971, 2021. [Online]. Available:
https://github.com/brianlarkins/saws.
[110] X10 (GitHub repository), commit hash: 5412ae0a0db1, IBM Corporation, 2020. [Online]. Available:
https://github.com/x10-lang/x10.
[111] X10 benchmarks (GitHub repository), commit hash: 2d6a59614ad9, IBM Corporation, 2020. [Online].
Available: https://github.com/x10-lang/x10-benchmarks.
[112] D. S. Hirschberg, “A linear space algorithm for computing maximal common subsequences,” Commun.
ACM, vol. 18, no. 6, pp. 341–343, 1975.
[113] R. A. Chowdhury and V. Ramachandran, “Cache-oblivious dynamic programming,” in Proceedingsof the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm, SODA ’06, Miami, Florida,
USA, 2006, pp. 591–600.
[114] Y. Tang, R. You, H. Kan, J. J. Tithi, P. Ganapathi, and R. A. Chowdhury, “Cache-oblivious wavefront:
Improving parallelism of recursive dynamic programming algorithms without losing cache-eﬃciency,”
in Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’15, San Francisco, California, USA, 2015, pp. 205–214.
[115] R. Chowdhury, P. Ganapathi, Y. Tang, and J. J. Tithi, “Provably eﬃcient scheduling of cache-oblivious
wavefront algorithms,” in Proceedings of the 29th ACM Symposium on Parallelism in Algorithms andArchitectures, SPAA ’17, Washington, District of Columbia, USA, 2017, pp. 339–350.
[116] K. Singer, Y. Xu, and I.-T. A. Lee, “Proactive work stealing for futures,” in Proceedings of the 24th ACM
SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’19, Washington,
District of Columbia, USA, 2019, pp. 257–271.
[117] G. E. Blelloch, J. T. Fineman, P. B. Gibbons, and H. V. Simhadri, “Scheduling irregular parallel
computations on hierarchical caches,” in Proceedings of the Twenty-Third Annual ACM Symposium on
Parallelism in Algorithms and Architectures, SPAA ’11, San Jose, California, USA, 2011, pp. 355–366.
[118] Z. Zhang, J. Savant, and S. Seidel, “A UPC runtime system based on MPI and POSIX threads,” inProceedings of the 14th Euromicro International Conference on Parallel, Distributed, and NetworkBased Processing, PDP ’06, Montbeliard-Sochaux, France, 2006, pp. 195–202.
[119] M. P. Ferguson and D. Buettner, “Caching puts and gets in a PGAS language runtime,” in Proceedingsof the 2015 9th International Conference on Partitioned Global Address Space Programming Models,
PGAS ’15, Washington, District of Columbia, USA, 2015, pp. 13–24.
[120] Q. Cai, W. Guo, H. Zhang, et al., “Eﬃcient distributed memory management with RDMA and caching,”
Proceedings of the VLDB Endowment, vol. 11, no. 11, pp. 1604–1617, 2018.
[121] J. Zhang, X. Yu, Z. Qi, and H. Guan, “Falcon: A timestamp-based protocol to maximize the cacheeﬃciency in the distributed shared memory,” in Proceedings of the 36th IEEE International Paralleland Distributed Processing Symposium, IPDPS ’22, Lyon, France, 2022, pp. 974–984.
[122] J. Bachan, S. Baden, D. Bonachea, et al., “UPC++ v1.0 programmer s guide, revision 2022.9.0,”
Lawrence Berkeley National Laboratory, USA, Tech. Rep. LBNL-2001479, 2022.

141References[123] M. D. Sinclair, J. Alsop, and S. V. Adve, “Eﬃcient GPU synchronization without scopes: Saying noto complex consistency models,” in Proceedings of the 48th International Symposium on Microarchitecture, MICRO-48, Waikiki, Hawaii, USA, 2015, pp. 647–659.
[124] G. E. Blelloch, P. B. Gibbons, and H. V. Simhadri, “Low depth cache-oblivious algorithms,” in Proceedings of the Twenty-Second Annual ACM Symposium on Parallelism in Algorithms and Architectures,
SPAA ’10, Thira, Santorini, Greece, 2010, pp. 189–199.
[125] R. A. Chowdhury, F. Silvestri, B. Blakeley, and V. Ramachandran, “Oblivious algorithms for multicoresand network of processors,” in Proceedings of the 24th IEEE International Parallel and Distributed
Processing Symposium, IPDPS ’10, Atlanta, Georgia, USA, 2010.
[126] R. A. Chowdhury, V. Ramachandran, F. Silvestri, and B. Blakeley, “Oblivious algorithms for multicoresand networks of processors,” Journal of Parallel and Distributed Computing, vol. 73, no. 7, pp. 911–
925, 2013.
[127] J. Liﬄander, S. Krishnamoorthy, and L. V. Kale, “Optimizing data locality for fork/join programs using
constrained work stealing,” in Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis, SC ’14, New Orleans, Los Angeles, USA, 2014,
pp. 857–868.
[128] Q. Chen, M. Guo, and H. Guan, “LAWS: Locality-aware work-stealing for multi-socket multi-corearchitectures,” in Proceedings of the 28th ACM International Conference on Supercomputing, ICS ’14,
Munich, Germany, 2014, pp. 3–12.
[129] Q. Chen and M. Guo, “Locality-aware work stealing based on online proﬁling and auto-tuning for multisocket multicore architectures,” ACM Transactions on Architecture and Code Optimization, vol. 12,
no. 2, 22:1–22:24, 2015.
[130] Q. Chen, M. Guo, and Z. Huang, “CATS: Cache aware task-stealing based on online proﬁling in
multi-socket multi-core architectures,” in Proceedings of the 26th ACM International Conference on
Supercomputing, ICS ’12, San Servolo Island, Venice, Italy, 2012, pp. 163–172.
[131] J.-N. Quintin and F. Wagner, “Hierarchical work-stealing,” in Proceedings of the 16th European
Conference on Parallel Processing, Euro-Par ’10, Ischia, Italy, 2010, pp. 217–229.
[132] G. E. Blelloch, R. A. Chowdhury, P. B. Gibbons, V. Ramachandran, S. Chen, and M. Kozuch,“Provably good multicore cache performance for divide-and-conquer algorithms,” in Proceedingsof the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’08, San Francisco,California, USA, 2008, pp. 501–510.
[133] H. V. Simhadri, G. E. Blelloch, J. T. Fineman, P. B. Gibbons, and A. Kyrola, “Experimental analysis ofspace-bounded schedulers,” in Proceedings of the 26th ACM Symposium on Parallelism in Algorithms
and Architectures, SPAA ’14, Prague, Czech Republic, 2014, pp. 30–41.
[134] H. V. Simhadri, G. E. Blelloch, J. T. Fineman, P. B. Gibbons, and A. Kyrola, “Experimental analysis ofspace-bounded schedulers,” ACM Transactions on Parallel Computing, vol. 3, no. 1, 8:1–8:27, 2016.
[135] L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, Classiﬁcation and Regression Trees. CRC
Press, 1984.
[136] K. Alsabti, S. Ranka, and V. Singh, “CLOUDS: A decision tree classiﬁer for large datasets,” inProceedings of the 4th International Conference on Knowledge Discovery and Data Mining, KDD’98, New York, New York, USA, 1998, pp. 2–8.

142

References[137] R. Jin and G. Agrawal, “Communication and memory eﬃcient parallel decision tree construction,” inProceedings of the 2003 SIAM International Conference on Data Mining, SDM ’03, San Francisco,California, USA, 2003, pp. 119–129.
[138] G. Ke, Q. Meng, T. Finley, et al., “LightGBM: A highly eﬃcient gradient boosting decision tree,” inProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS
’17, Long Beach, California, USA, 2017, pp. 3149–3157.
[139] B. Alpern, L. Carter, and J. Ferrante, “Modeling parallel computers as memory hierarchies,” inProceedings of Workshop on Programming Models for Massively Parallel Computers, PMMP ’93,
Berlin, Germany, 1993, pp. 116–123.
[140] Oakbridge-CX supercomputer system, The University of Tokyo, 2019. [Online]. Available: https://www.cc.u-tokyo.ac.jp/en/supercomputer/obcx/service/.
[141] Intel 64 and IA-32 architectures software developer’s manual – volume 3B: system programming guide,part 2, Intel Corporation, 2020.
[142] Intel Xeon processor scalable memory family uncore performance monitoring, Intel Corporation, 2017.
[143] D. Dua and C. Graﬀ, UCI machine learning repository, 2017. [Online]. Available: http://archive.
ics.uci.edu/ml.
[144] P. Baldi, P. Sadowski, and D. Whiteson, “Searching for exotic particles in high-energy physics withdeep learning,” Nature Communications, vol. 5, no. 4308, pp. 1–9, 2014.
[145] M. Becker and M. Teschner, “Weakly compressible SPH for free surface ﬂows,” in Proceedings ofthe 2007 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’07, San Diego,
California, USA, 2007, pp. 209–217.
[146] M. Iwasawa, A. Tanikawa, N. Hosono, K. Nitadori, T. Muranushi, and J. Makino, “Implementation andperformance of FDPS: A framework for developing parallel particle simulation codes,” Publicationsof the Astronomical Society of Japan, vol. 68, no. 4, 54:1–54:22, 2016.
[147] J. Barnes and P. Hut, “A hierarchical 𝑂 (𝑁 log 𝑁) force-calculation algorithm,” Nature, vol. 324,
pp. 446–449, 1986.
[148] SQUID (supercomputer for quest to unsolved interdisciplinary datascience), Cybermedia Center,
Osaka University, 2021. [Online]. Available: http://www.hpc.cmc.osaka-u.ac.jp/en/squid/.
[149] H. Sundar, D. Malhotra, and G. Biros, “HykSort: A new variant of hypercube quicksort on distributed
memory architectures,” in Proceedings of the 27th International ACM Conference on International
Conference on Supercomputing, ICS ’13, Eugene, Oregon, USA, 2013, pp. 293–302.
[150] H. Sundar, Utah sorting library, 2016. [Online]. Available: https://github.com/hsundar/usort.
[151] J. Nelson, B. Holt, B. Myers, et al., “Grappa: A latency-tolerant runtime for large-scale irregularapplications,” in Proceedings of the First International Workshop on Rack-Scale Computing, WRSC
’14, Amsterdam, The Netherlands, 2014, pp. 1–7.
[152] R. Yokota, L. A. Barba, T. Narumi, and K. Yasuoka, “Petascale turbulence simulation using a highly
parallel fast multipole method on GPUs,” Computer Physics Communications, vol. 184, no. 3, pp. 445–
455, 2013.

143

References[153] K. Taura, J. Nakashima, R. Yokota, and N. Maruyama, “A task parallel implementation of fast multipole
methods,” in 2012 SC Companion: High Performance Computing, Networking Storage and Analysis–
Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems, ScalA’ 12, Salt Lake
City, Utah, USA, 2012, pp. 617–625.
[154] R. Yokota and L. Barba, GitHub repository: exafmm/exafmm-beta, 2020. [Online]. Available: https://github.com/exafmm/exafmm-beta.
[155] A. Danial, GitHub repository: AlDanial/cloc, 2018. [Online]. Available: https : / / github . com /AlDanial/cloc.
[156] H. C. Plummer, “On the problem of distribution in globular star clusters,” Monthly Notices of the Royal
Astronomical Society, vol. 71, pp. 460–470, 1911.
[157] M. Dobson, Z. Shen, G. E. Blelloch, et al., Scaling graph-based ANNS algorithms to billion-size
datasets: A comparative analysis, 2023. arXiv: 2305.04359 [cs.IR].
[158] Y. A. Malkov and D. A. Yashunin, “Eﬃcient and robust approximate nearest neighbor search using hierarchical navigable small world graphs,” IEEE transactions on pattern analysis and machine
intelligence, vol. 42, no. 4, pp. 824–836, 2018.
[159] J. V. Muñoz, M. A. Gonçalves, Z. Dias, and R. d. S. Torres, “Hierarchical clustering-based graphs forlarge scale approximate nearest neighbor search,” Pattern Recognition, vol. 96, no. 106970, pp. 1–10,
2019.
[160] M. Aumüller, Big ANN benchmarks, 2021. [Online]. Available: https : / / github . com / harsha simhadri/big-ann-benchmarks.
[161] L. Dhulipala, G. E. Blelloch, and J. Shun, “Theoretically eﬃcient parallel graph algorithms can be fastand scalable,” ACM Transactions on Parallel Computing, vol. 8, no. 1, 4:1–4:70, 2021.
[162] L. Dhulipala, G. E. Blelloch, and J. Shun, GBBS: Graph based benchmark suite, 2018. [Online].
Available: https://github.com/ParAlg/gbbs.
[163] K. Lakhotia, R. Kannan, and V. Prasanna, “Accelerating PageRank using partition-centric processing,”
in Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC ’18, Boston, MA,
USA, 2018, pp. 427–440.
[164] K. Lakhotia, R. Kannan, S. Pati, and V. Prasanna, “GPOP: A scalable cache- and memory-eﬃcient
framework for graph processing over parts,” ACM Transactions on Parallel Computing, vol. 7, no. 1,
7:1–7:24, 2020.
[165] D. Chakrabarti, Y. Zhan, and C. Faloutsos, “R-MAT: A recursive model for graph mining,” in Proceedings of the 2004 SIAM International Conference on Data Mining, SDM ’04, Orlando, Florida,
USA, 2004, pp. 442–446.
[166] H. Kwak, C. Lee, H. Park, and S. Moon, “What is Twitter, a social network or a news media?” In
Proceedings of the 19th International Conference on World Wide Web, WWW ’10, Raleigh, North
Carolina, USA, 2010, pp. 591–600.
[167] F. McSherry, M. Isard, and D. G. Murray, “Scalability! but at what COST?” In Proceedings of the
15th Workshop on Hot Topics in Operating Systems, HotOS XV, Kartause Ittingen, Switzerland, 2015.

144

References[168] X. Zhu, W. Chen, W. Zheng, and X. Ma, “Gemini: A computation-centric distributed graph processing system,” in Proceedings of the 12th USENIX Symposium on Operating Systems Design and
Implementation, OSDI ’16, Savannah, GA, USA, 2016, pp. 301–316.
[169] D. Hendler and N. Shavit, “Non-blocking steal-half work queues,” in Proceedings of the Twenty-First
Annual Symposium on Principles of Distributed Computing, PODC ’02, Monterey, California, USA,2002, pp. 280–289.
[170] T.-T. Vu and B. Derbel, “Link-heterogeneous work stealing,” in Proceedings of the 14th IEEE/ACM
International Symposium on Cluster, Cloud and Grid Computing, CCGrid ’14, Chicago, Illinois, USA,2014, pp. 354–363.
[171] R. Nakashima, H. Yoritaka, M. Yasugi, T. Hiraishi, and S. Umatani, “Extending a work-stealing
framework with priorities and weights,” in Proceedings of the 2019 IEEE/ACM 9th Workshop on
Irregular Applications: Architectures and Algorithms, IA3 ’19, Denver, Colorado, USA, 2019, pp. 9–16.
[172] Y. Guo, J. Zhao, V. Cave, and V. Sarkar, “SLAW: A scalable locality-aware adaptive work-stealingscheduler,” in Proceedings of the 24th IEEE International Symposium on Parallel and Distributed
Processing Symposium, IPDPS ’10, Atlanta, GA, USA, 2010.
[173] K. Li and P. Hudak, “Memory coherence in shared virtual memory systems,” ACM Transactions on
Computer Systems, vol. 7, no. 4, pp. 321–359, 1989.
[174] J. K. Bennett, J. B. Carter, and W. Zwaenepoel, “Munin: Distributed shared memory based on typespeciﬁc memory coherence,” in Proceedings of the Second ACM SIGPLAN Symposium on Principles
and Practice of Parallel Programming, PPOPP ’90, Seattle, Washington, USA, 1990, pp. 168–176.
[175] H.-R. Chuang, R. Lyerly, S. Lankes, and B. Ravindran, “Scaling shared memory multiprocessing
applications in non-cache-coherent domains,” in Proceedings of the 13th ACM International Systems
and Storage Conference, SYSTOR ’20, Haifa, Israel, 2020, pp. 13–24.
[176] A. Morari, A. Tumeo, D. Chavarrı́a-Miranda, O. Villa, and M. Valero, “Scaling irregular applications
through data aggregation and software multithreading,” in Proceedings of the 28th IEEE International
Parallel and Distributed Processing Symposium, IPDPS ’14, Phoenix, Arizona, USA, 2014, pp. 1126–
1135.
[177] J. Schuchart and J. Gracia, “Global task data-dependencies in PGAS applications,” in High Performance
Computing: the 34th International Conference, ISC High Performance 2019, ISC ’19, Frankfurt/Main,
Germany, 2019, pp. 312–329.
[178] K. Tsugane, J. Lee, H. Murai, and M. Sato, “Multi-tasking execution in PGAS language XcalableMP andcommunication optimization on many-core clusters,” in Proceedings of the International Conference
on High Performance Computing in Asia-Paciﬁc Region, HPC Asia 2018, Chiyoda, Tokyo, Japan,
2018, pp. 75–85.
[179] M. Bauer, S. Treichler, E. Slaughter, and A. Aiken, “Legion: Expressing locality and independencewith logical regions,” in Proceedings of the International Conference on High Performance Computing,
Networking, Storage and Analysis, SC ’12, Salt Lake City, Utah, USA, 2012, 66:1–66:11.
[180] G. Bosilca, A. Bouteiller, A. Danalis, M. Faverge, T. Herault, and J. J. Dongarra, “PaRSEC: Exploiting
heterogeneity to enhance scalability,” Computing in Science & Engineering, vol. 15, no. 6, pp. 36–45,
2013.

145

References[181] C. Augonnet, S. Thibault, R. Namyst, and P.-A. Wacrenier, “StarPU: A uniﬁed platform for taskscheduling on heterogeneous multicore architectures,” in Proceedings of the 15th International European Conference on Parallel and Distributed Computing, Euro-Par ’09, Delft, The Netherlands, 2009,
pp. 863–874.
[182] J. Bueno, L. Martinell, A. Duran, et al., “Productive cluster programming with OmpSs,” in Proceedingsof the 17th International European Conference on Parallel and Distributed Computing, Euro-Par ’11,
Bordeaux, France, 2011, pp. 555–566.
[183] J. A. Mena, O. Shaaban, V. Beltran, P. Carpenter, E. Ayguade, and J. Labarta, “OmpSs-2@Cluster:Distributed memory execution of nested OpenMP-style tasks,” in Proceedings of the 28th International
European Conference on Parallel and Distributed Computing, Euro-Par ’22, Glasgow, Scotland, UK,2022, pp. 319–334.
[184] A. Basumallik and R. Eigenmann, “Optimizing irregular shared-memory applications for distributedmemory systems,” in Proceedings of the Eleventh ACM SIGPLAN Symposium on Principles andPractice of Parallel Programming, PPoPP ’06, New York, New York, USA, 2006, pp. 119–128.
[185] O. Kwon, F. Jubair, R. Eigenmann, and S. Midkiﬀ, “A hybrid approach of OpenMP for clusters,” in Proceedings of the 17th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming,
PPoPP ’12, New Orleans, Louisiana, USA, 2012, pp. 75–84.
[186] G. E. Blelloch and P. B. Gibbons, “Eﬀectively sharing a cache among threads,” in Proceedings of
the Sixteenth Annual ACM Symposium on Parallelism in Algorithms and Architectures, SPAA ’04,
Barcelona, Spain, 2004, pp. 235–244.
[187] G. J. Narlikar and G. E. Blelloch, “Space-eﬃcient scheduling of nested parallelism,” ACM Transactions
on Programming Languages and Systems, vol. 21, no. 1, pp. 138–173, 1999.
[188] G. J. Narlikar, “Scheduling threads for low space requirement and good locality,” Theory of Computing
Systems, vol. 35, no. 2, pp. 151–187, 2002.
[189] G. E. Blelloch, P. B. Gibbons, and H. V. Simhadri, “Provably eﬃcient scheduling of dynamicallyallocating programs on parallel cache hierarchies,” in Proceedings of the 24th IEEE International
Conference on High Performance Computing, HiPC ’17, Jaipur, India, 2017, pp. 124–133.
[190] A. Robison, M. Voss, and A. Kukanov, “Optimization via reﬂection on work stealing in TBB,” inProceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, IPDPS’08, Miami, Florida, USA, 2008.
[191] J. Deters, J. Wu, Y. Xu, and I.-T. A. Lee, “A NUMA-aware provably-eﬃcient task-parallel platform
based on the work-ﬁrst principle,” in Proceedings of the 2018 IEEE International Symposium on
Workload Characterization, IISWC ’18, Raleigh, North Carolina, USA, 2018, pp. 59–70.
[192] A. Drebes, K. Heydemann, N. Drach, A. Pop, and A. Cohen, “Topology-aware and dependence-aware
scheduling and memory allocation for task-parallel languages,” ACM Transactions on Architecture andCode Optimization, vol. 11, no. 3, 30:1–30:25, 2014.
[193] A. Drebes, A. Pop, K. Heydemann, A. Cohen, and N. Drach, “Scalable task parallelism for NUMA:
A uniform abstraction for coordinated scheduling and memory management,” in Proceedings of the2016 International Conference on Parallel Architectures and Compilation, PACT ’16, Haifa, Israel,
2016, pp. 125–137.

146

References[194] J. Maglalang, S. Krishnamoorthy, and K. Agrawal, “Locality-aware dynamic task graph scheduling,”
in Proceedings of the 46th International Conference on Parallel Processing, ICPP ’17, Bristol, UK,2017, pp. 70–80.
[195] V. Kumar, “PuﬀerFish: NUMA-aware work-stealing library using elastic tasks,” in Proceedings ofthe 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics,
HiPC ’20, Pune, India, 2020, pp. 251–260.
[196] A. Pop and A. Cohen, “OpenStream: Expressiveness and data-ﬂow compilation of openmp streaming
programs,” ACM Transactions on Architecture and Code Optimization, vol. 9, no. 4, 53:1–53:25, 2013.
[197] J. Liﬄander, S. Krishnamoorthy, and L. V. Kale, “Steal tree: Low-overhead tracing of work stealingschedulers,” in Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design
and Implementation, PLDI ’13, Seattle, Washington, USA, 2013, pp. 507–518.
[198] Y. Wang, Y. Zhang, Y. Su, et al., “An adaptive and hierarchical task scheduling scheme for multi-coreclusters,” Parallel Computing, vol. 40, no. 10, pp. 611–627, 2014.
[199] T. B. Schardl, B. C. Kuszmaul, I.-T. A. Lee, W. M. Leiserson, and C. E. Leiserson, “The Cilkprof
scalability proﬁler,” in Proceedings of the 27th ACM Symposium on Parallelism in Algorithms andArchitectures, SPAA ’15, Portland, Oregon, USA, 2015, pp. 89–100.
[200] A. Huynh, D. Thain, M. Pericàs, and K. Taura, “DAGViz: A DAG visualization tool for analyzingtask-parallel program traces,” in Proceedings of the 2nd Workshop on Visual Performance Analysis,
VPA ’15, Austin, Texas, 2015, 3:1–3:8.

147