Homomorphism-based
Structured Parallel Programming

準同形に基づいた構造化並列プログラミングに関する研究

江本

健斗



Committee
Professor

Masato Takeichi

Professor
Professor
Professor
Associate Professor
Associate Professor

Kazuo Murota
Masaaki Sugihara
Zhenjiang Hu
Kenjiro Taura
Shin-Cheng Mu

Acknowledgments
I would like to acknowledge all those people who kindly supported and assisted meduring my research.
First of all, I would like to give my heartfelt thanks to Professor Masato Takeichi,
who had been my supervisor at my doctoral course of the graduate school of the
University of Tokyo, for his kind and helpful encouragement and guidance during
my research.
I am also full of gratitude to Professor Zhenjiang Hu for giving me many valuablesuggestions and great encouragement through a lot of helpful discussions.
Special thanks go Dr. Kiminori Matsuzaki, Dr. Kazuhiko Kakehi, and Professor
Hideya Iwasaki for their insightful comments and kind encouragements to carry out
this research.
I would like to thank Dr. Guy Lewis Steele Jr. and Dr. Jan-Willem Maessen forvaluable suggestions through a lot of fruitful discussions.
I also thank all members of the laboratory. Especially, I thank my clever colleagues, Dr. Akimasa Morihata and Dr. Kazutaka Matsuda, for various kinds ofsuggestions during my laboratory life.
The thesis is composed of several research papers I have written and submitted tojournals, conferences or workshops. I acknowledge several anonymous referees who
gave me many helpful comments during the process of the research. The committee
members gave many helpful comments to this thesis.

Abstract
In recent years, parallel programming has become an essential task for programmers, but it is still a very difficult task for most programmers. There are two
difficult problems in parallel programming: difficulty of making (designing) efficientparallel algorithms, and difficulty of implementing the parallel algorithms. The former problem is caused by lack of useful characterization of parallel algorithms andsystematic methods for deriving efficient parallel algorithms. The latter problem iscaused by the need to implement the extra considerations, such as distribution oftasks among processors, communication of data among processors, and synchronization of progress of the tasks, which are not required in sequential programming.
Structured parallel programming, also known as skeletal parallel programming,has been proposed as one promising methodology to solve the problems of parallel
programming. In this methodology, programmers build parallel programs (algorithms) by composing skeletons, i.e., ready-made parallel components that haveefficient parallel implementation on various parallel architectures. This approachhas the advantage that user programmers can compose parallel programs easily
without considering difficult low-level parallelism, because low-level parallelism iscompletely concealed in skeletons. Once we compose parallel algorithms describedwith skeletons, we can straightforwardly implement them with skeleton implementation, which is usually provided as either libraries on existing programming languages
or new programming languages including skeletons as their primitive constructs.
One difficult problem in skeletal parallel programming is to provide a methodology to develop efficient parallel algorithms. Especially, optimization of skeleton
compositions to improve their efficiency is very important, because it provides us
with a systematic way to compose efficient parallel algorithms from naively composed skeleton programs. In general, naively composed skeleton programs sufferfrom inefficiency caused by redundancy of compositions, such as redundant intermediate data structures communicated between composed skeletons.
The thesis tackles the problem by homomorphism-based design of parallel skeletons that have good properties to develop efficient parallel algorithms by fusion.
Fusion optimization is one effective optimization to derive efficient parallel algorithms. It removes redundancy of compositions by fusing consecutive skeletons in thecompositions. Therefore, we can develop efficient parallel algorithms from naively
composed skeleton programs by applying fusions to the compositions repeatedly.
The following are three important contributions of this thesis.

viii
The first contribution is the design of parallel skeletons based on homomorphisms, which provides skeletons with both good composability with each other andnice algebraic properties to optimize their compositions. Especially, design of skeletons for two-dimensional arrays remained as a challenging problem. Having designed
the skeletons, we have be able to develop parallel programs to perform computation on various data structures, and to discuss structured parallel programming andoptimization of structured parallel programs.
The second contribution is development of fusion optimizations for structured
parallel programs. Parallel programs written with skeletons often suffers from theinefficiency problem caused by redundant generation of intermediate data structures among skeleton compositions. We have studied fusion optimizations of skeletons based on the fusibility of homomorphisms, which can remove intermediate
data structures by fusing consecutive skeletons. Also, we have demonstrated thepower of fusion optimization by a derivation of a non-trivial efficient algorithm forthe maximum rectangle sum problem, giving a strategy to develop efficient skeletons programs using the fusion optimization by hand. Moreover, we have studied
domain-specific fusion optimizations of skeleton programs for computation involving
neighbor elements, which has been formalized as nested reductions.
The third contribution is implementation of the skeletons and the optimization
mechanisms on C++ and Fortress. We have successfully implemented the designed
skeletons efficiently on parallel machines, with systems for domain-independent anddomain-specific optimizations. Moreover, we have shown that the optimizations
can be implemented at library-level, owing to the homomorphism-based design ofskeletons.

Contents
1 Introduction
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Homomorphism-based Structured Parallel Programming . . . . . . .
1.3 Contributions and Organizations of the Thesis . . . . . . . . . . . . .

1
1
2
8

2 Algebras and Homomorphisms for Data Structures
2.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Base Theory of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Algebra of Lists . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 List Homomorphism . . . . . . . . . . . . . . . . . . . . . . .
2.3 Base Theory of Two-Dimensional Arrays . . . . . . . . . . . . . . . .
2.3.1 Abide-tree Algebra for Two-Dimensional Arrays . . . . . . . .
2.3.2 Abide-tree Homomorphism . . . . . . . . . . . . . . . . . . . .
2.4 Base Theory of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Algebra of Trees . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Tree Homomorphisms . . . . . . . . . . . . . . . . . . . . . .
2.5 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11
11
14
14
15
16
16
18
19
19
20
22

3 Homomorphism-based Design of Parallel Skeletons25
3.1 Homomorphism-based Parallel Skeletons for Lists . . . . . . . . . . . 25
3.1.1 Definitions of Parallel Skeletons . . . . . . . . . . . . . . . . . 25
3.1.2 Example Programs and Variants of Skeletons . . . . . . . . . . 27
3.2 Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays 29
3.2.1 Definitions of Parallel Skeletons . . . . . . . . . . . . . . . . . 29
3.2.2 Example Complex Programs Described with Parallel Skeletons 39
3.3 Homomorphism-based Parallel Skeletons for Trees . . . . . . . . . . . 43
4 Fusion Optimizations of Parallel Skeletons
4.1 Useful Fusion Laws for Skeletons on Lists . . . . . . . . . . . . . . . .
4.1.1 Fusion of List Homomorphism . . . . . . . . . . . . . . . . . .
4.1.2 Fusion Laws for Skeletons . . . . . . . . . . . . . . . . . . . .
4.1.3 Accumulate-buildJ Fusion . . . . . . . . . . . . . . . . . . . .
4.2 Useful Fusion Laws for Skeletons on Two-dimensional Arrays . . . . .

47
48
48
48
49
50

x

4.3

4.4

4.2.1 Fusion of Homomorphism for Two-dimensional Arrays . . . .
4.2.2 Fusion Laws of Skeleton Compositions . . . . . . . . . . . . .
Developing Efficient Programs with Parallel Skeletons . . . . . . . . .
4.3.1 Almost-homomorphism . . . . . . . . . . . . . . . . . . . . . .
4.3.2 Two Theorems for Deriving Efficient Parallel Programs . . . .
4.3.3 A Strategy for Deriving Efficient Parallel Programs . . . . . .
4.3.4 Deriving an Efficient Parallel Program for Maximum Rectangle Sum Problem . . . . . . . . . . . . . . . . . . . . . . . . .
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51
52
53
53
54
57
57
71

5 Domain-Specific Optimization for Skeleton Programs
73
5.1 General Strategy for Domain-Specific Fusion Optimization . . . . . . 74
5.2 Computation Involving a Finite Number of Neighbor Elements . . . . 74
5.2.1 Target Skeleton Composition Patterns . . . . . . . . . . . . . 75
5.2.2 Normal Form for the Domain . . . . . . . . . . . . . . . . . . 77
5.2.3 Fusion Rules for Transformation to a Normal Form . . . . . . 81
5.2.4 Parallel Implementation of Normal Form . . . . . . . . . . . . 85
5.2.5 Expansion of Target Programs . . . . . . . . . . . . . . . . . . 91
5.3 Nested Reductions: Involving an Infinite Number of Neighbor Elements 96
5.3.1 Generate-and-test Specifications: General Forms of Nested
Reductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.3.2 Functions to Generate Nested Data Structures . . . . . . . . . 97
5.3.3 Useful Properties on Predicates . . . . . . . . . . . . . . . . . 110
5.3.4 Optimization Theorems for the Nested Reductions . . . . . . . 112
5.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6 Implementation of Skeletons and Optimizations
139
6.1 Implementation of Parallel Skeletons on Lists . . . . . . . . . . . . . 139
6.2 Implementation of Parallel Skeletons on Two-dimensional Arrays . . . 141
6.2.1 Implementation of Simple Parallel Skeletons . . . . . . . . . . 142
6.2.2 Implementation of Scan Skeleton . . . . . . . . . . . . . . . . 143
6.2.3 Refined Implementation for Nested Use of Skeletons . . . . . . 155
6.2.4 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . 157
6.3 Implementation of Fusion Optimizations . . . . . . . . . . . . . . . . 158
6.3.1 A Simple System for Domain-Independent Fusion Optimization158
6.3.2 A Simple System for Domain-Specific Fusion Optimization . . 161
6.3.3 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . 165
6.4 Libraries with Optimization Capabilities . . . . . . . . . . . . . . . . 169
6.4.1 General Design of Growable Optimizing Libraries . . . . . . . 170
6.4.2 Parallel Programming Language Fortress . . . . . . . . . . . . 172
6.4.3 Generators for Reductions in Fortress . . . . . . . . . . . . . . 176
6.4.4 Generators-of-generators: the Core of Nested Reductions . . . 180
6.4.5 Design and Implementation of the Optimizing GoG Library . 181

Contents

6.5

xi

6.4.6 Programming with the Library . . . . . . . . . . . . . . . . . 188
6.4.7 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . 191
6.4.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

7 Conclusion
199
7.1 Summary of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 199
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Bibliography

203

A Auxiliary Rules for Skeletons on Two-dimensional Arrays213

B Complete Proof of Theorem 5.1

225

C Auxiliary Rules for the Proof of Theorem 5.44

231

D Dilation of Rects

235

Chapter 1
Introduction
1.1

BackgroundIn recent years, parallel programming has become an essential task for programmers,
but it is still a very difficult task for most programmers. Most of the latest computers
have become parallel computers equipped with multi-core CPUs, and small scale PC
clusters have become cheap to be used by individual programmers. At the same time,
problems that we want to solve with computers have become bigger and heavier, andunsolvable in reasonable time and space without power of parallel computers. Thissituation strongly enforces parallel programming upon programmers for maximum
use of these parallel computers. Parallel programming is, however, a very difficult
task for most programmers, because programmers must take into account extra
considerations such as distribution of tasks among processors, communication ofdata among processors, and synchronization of progress of the tasks.
There are two difficult problems in parallel programming: difficulty of making(designing) efficient parallel algorithms, and difficulty of implementing the parallel
algorithms. The former problem is caused by lack of useful characterization ofparallel algorithms and systematic methods for deriving efficient parallel algorithms.
The latter problem is caused by the need to implement the extra considerations, suchas distribution of tasks among processors, communication of data among processors,and synchronization of progress of the tasks, which are not required in sequential
programming.
Structured parallel programming, also known as skeletal parallel programming,has been proposed as one promising methodology to solve the problems of parallel
programming [RG02,Col02,Col89]. In this methodology, programmers build parallel
programs (algorithms) by composing skeletons, i.e., ready-made parallel componentsthat have efficient parallel implementation on various parallel architectures. Thisapproach has two major advantages. The first advantage is that user programmers can compose parallel programs easily without considering difficult low-level
parallelism, because low-level parallelism is completely concealed in skeletons. The
other advantage is that the composed parallel programs are easy to understand and2

1. Introduction

maintain, since they are structured by skeletons. In other words, skeletal parallel
programming enables us to make parallel programs in a sequential style.
Once we compose parallel algorithms described with skeletons, we can straightforwardly implement them with skeleton implementation, which is usually provided
as either libraries on existing programming languages or new programming languages including skeletons as their primitive constructs. Much research has been
done on skeletal parallel programming, and many skeleton libraries and systems
have been proposed, which includes P3L [DPP97, Pel98], SCL [DFH+ 93, DGTY95],
eSkel [Col04, BCHG05], MuesLi [Kuc02], QUAFF [FSCL06], and SkeTo [MIEH06].
One difficult problem in skeletal parallel programming is to provide a methodology to develop efficient parallel algorithms. Especially, optimization of skeleton
compositions to improve their efficiency is very important, because it provides us
with a systematic way to compose efficient parallel algorithms from naively composed skeleton programs. In general, naively composed skeleton programs sufferfrom inefficiency caused by redundancy of compositions, such as redundant intermediate data structures communicated between composed skeletons. This inefficiency
problem is an essential problem in compositional style programming of skeletal parallel programming.
The thesis tackles the problem by homomorphism-based design of parallel skeletons that have good properties to develop efficient parallel algorithms by fusion.
Fusion optimization is one effective optimization to derive efficient parallel algorithms. It removes redundancy of compositions by fusing consecutive skeletons in thecompositions. Therefore, we can develop efficient parallel algorithms from naively
composed skeleton programs by applying fusions to the compositions repeatedly.

1.2

Homomorphism-based
Structured Parallel Programming

We will briefly review our homomorphism-based structured parallel programming.
Let’s consider the following problem as an example: given a two-dimensional
array, compute the maximum of sums of all rectangle areas in the array. This
problem is called the maximum rectangle (or sub-array) sum problem, and important
as a sort of data mining and pattern matching of two dimensional [Ben84a, Ben84b,
Tak02, HIT97]. For example, for the following two-dimensional array
3 −1
4 −1 −5 1 −4 −1
5 −3 
−4
1
5
3
1


the maximum rectangle sum is 15, which is the sum of bold numbers.
Let’s start at designing the algorithm using skeletons. We are provided with the31.2. Homomorphism-based Structured Parallel Programming
following skeletons to manipulate two-dimensional arrays.

x11 x12 · · · x1n
f x11 f x12 · · · f x1nx  x21 x22 · · · x2n  f x21 f x22 · · ·
f2n




map f  ..
=  ..
..
..
.. 
.. 
.
.
.
.
 .
 .
.
.
.
.
. 
. 
xm1 xm2 · · · xmn
f xm1 f xm2 · · · f xmn


x11
 x21

reduce (⊕, ⊗)  ..
 .

x12
x22
..
.

···
···
...

xm1 xm2 · · ·



x11
 x21

scan (⊕, ⊗)  ..
 .

x12
x22
..
.


x1n
x2n 

..  =
. 

xmn


x1n
x2n 

.. 
. 

···
···
...

xm1 xm2 · · ·(x11 ⊗ x12 ⊗ · · · ⊗ x1n )⊕(x21 ⊗ x22 ⊗ · · · ⊗ x2n )⊕
..
.

xmn

(xm1 ⊗ xm2 ⊗ · · · ⊗ xmn )


y11
 y21

=  ..
 .

y12
y22
..
.

···
···
...

ym1 ym2 · · ·where yij =


y1n
y2n 

.. 
. 

ymn

(x11 ⊗ x12 ⊗ · · · ⊗ x1j )⊕(x21 ⊗ x22 ⊗ · · · ⊗ x2j )⊕..
.
(xi1 ⊗ xi2 ⊗ · · · ⊗ xij )



x11
 x21

scanr (⊕, ⊗)  ..
 .

x12
x22
..
.

···
···
...

xm1 xm2 · · ·


x1n
x2n 

.. 
. 

xmn



z11
 z21

=  ..
 .

where zij =

z12
z22
..
.

···
···
...

zm1 zm2 · · ·


z1n
z2n 

.. 
. 

zmn

(xij ⊗ · · · ⊗ xin )⊕..
.
(xmj ⊗ · · · ⊗ xmn )The skeleton map applies the given function f to every element of the given twodimensional array. The skeleton reduce collapses a two-dimensional array into avalue using two associative binary operators ⊕ and ⊗. In other words, the skeleton
reduce takes a sum of the given array with the given operators, in which the sum ofvertical direction is computed with the first operator ⊕, and the sum of horizontal
direction is computed with the second operator ⊗. The skeletons scan returns anarray of which elements are partial results of applying reduce to the array, i.e., valuesof applying reduce to sub-arrays from the top-left to the bottom-right. The skeleton
scanr works similarly in the reverse direction. Those skeletons are designed basedon homomorphism as briefly mentioned later.

4

1. Introduction
We can easily compose these skeletons to solve the problem as follows.

◦
−
◦ −

−
◦

mrs = max ◦ map sum ◦ rects ′wheremax = reduce (↑, ↑)sum = reduce (+, +)rects ′ = reduce (−◦, ) ◦ map TLs ◦ BRs
TLs= scan (−◦, ) ◦ map | · |BRs = scanr (−◦, ) ◦ map | · |

−
◦

−
◦

−
◦

Here, ◦ is a binary operator for function compositions, ↑ is a binary operator returning the bigger operand, and three operators | · |, −◦, and are used to construct
two-dimensional arrays: Operator | · | makes a singleton array of the given element,−◦ builds a bigger array from two arrays of the same width by stacking them, andcombines two arraysof the same height horizontally. For example, 2 ↑ 5 = 5,






1 2
1 2 −
5 6 = 1 2 5 6 . The function rects ′, and 1 2
◦ 5 6 =
5 6
generates all possible rectangles of the given array, using TLs and BRs to generate
top-left and bottom-rightrectangles, respectively. For example, applying TLs, BRs,

1 2
, we get the following results.
and rects ′ to
5 6

 


1
1
2

  
1 2
1 2 
=  1
TLs
5 6
5  5 6


2
1 2
1 2

6 
5 6
=
BRs
5 6
6

 5 6
1
1
2
 2

  

 1
2 
1 2
1 2

= 
rects ′

 5
6
5
6
5 6



5
5 6
6

−
◦

The computation of mrs = max ◦ map sum ◦ rects ′ is very clear: enumerating all
possible rectangles by rects ′ , then computing sums for all rectangles by map sum, andfinally returning the maximum value by max as the result of the whole computation.
Now, we have a clear naive algorithm to solve the problem. However, it seems
inefficient in the sense that it needs to execute O(n6 ) addition operations for the
input of n × n array.
The next step is to get an efficient algorithm from the naive one by fusions. To
this end, let’s see the basis of homomorphism-based skeletons for fusion optimization.
We will start at defining algebras of the target data structures. As an algebra to
represent two-dimensional arrays, we will borrow abide-tree algebra from the theoryof Constructive Algorithmics for sequential programming [Bir88]. The algebra consists of three constructors (operators): | · | (singleton), −◦ (above), and (beside).

51.2. Homomorphism-based Structured Parallel Programming

−
◦

−
◦

−
◦

On this algebra, any two-dimensional array is constructed by the three construc
1 2
is constructed by either
tors. For example, 2 × 2 two-dimensional array
5 6◦ (| · | 5 | · | 6) or (| · | 1−
◦ | · | 5) (| · | 2−
◦ | · | 6). It is worth noting(| · | 1 | · | 2)−that we think that these two constructions are equivalent (this equivalence is called
abide-property), and use this “freedom” to capture the maximum parallelism, which
will be used to get efficient parallel implementation. It is also worth noting that we
use one algebra for one data structure. For example, we use another algebra calledjoin-list for one-dimensional data arrays.
The next step to our fusion optimization is to define homomorphism of thealgebra, by which our fusible skeletons are structured. Homomorphisms are recursivefunctions of specific forms; their computation structures and their processing data
structures are closely related to each other. For two-dimensional arrays, we will havethe following homomorphism h, which is parameterized with a function f and two
associative binary operators ⊕ and ⊗ that have the abide-property.

−◦h (| · |a) = f ah (x−◦ y) = h x ⊕ h yh (x y) = h x ⊗ h y

−
◦

−
◦

−
◦

For notational convenience, we write ([f, ⊕, ⊗]) to denote h. Intuitively, a homomorphism ([f, ⊕, ⊗]) is a function to replace the constructors | · |, −◦, and in the input by
f , ⊕, and ⊗, respectively. It is worth noting that homomorphism has efficient (balanced) parallel implementation by naive divide-and-conquer parallel computation,in which the freedom of the algebra plays an important role for balancing.
Actually, the skeletons shown above are homomorphisms. For example, map f =([| · | ◦ f,−◦, ]), and reduce (⊕, ⊗) = ([id ,−◦, ]), where id is the identity function. Also,
scan and scanr are special cases of homomorphisms, but we omit them here because
they are too complicated to be shown here.
The reason why we structure our skeletons by homomorphism is that homomorphism has the following nice property for fusion optimization. Given a function g andhomomorphism ([f, ⊕, ⊗]), the composition g ◦ ([f, ⊕, ⊗]) is fused into another homomorphism ([g ◦ f, ⊙, ⊖]), when the following equations hold: g (x ⊕ y) = g x ⊙ g y
and g (x ⊗ y) = g x ⊖ g y. This property is very useful, because we can repeatedly
fuse functions into a homomorphism.
Our skeletons have good fusibility based on the fusibility of homomorphism. For
example, we can fuse two maps into one map as follows.

−
◦

−
◦

−
◦

−
◦

−
◦

map f ◦ map g= { map g is homomorphism ([| · | ◦ g,−◦, ]) }
map
f
◦
([|
·
|
◦
g,−
◦,
]
)


map f (x−◦ y) = map f x−
◦ map f y=homomorphism fusion with
map f (x y) = map f x map f y([ map f ◦ | · | ◦ g,−
◦, ])



6

1. Introduction{ simplification (map f ◦ | · | ◦ g = | · | ◦ f ◦ g) }([| · | ◦ f ◦ g,−◦, ])= { map is homomorphism }map (f ◦ g)−◦=Here, each use of braces ({ }) shows a reason of an equation.
Now, let’s return to our example. Using fusions of skeletons repeatedly, we canget the following optimized algorithm for mrs. Here, the complex composition ofskeletons is fused into only one homomorphism, and it is split into two skeletons
reduce and map. We will omit the details here (please see Section 4.3.4 for details).
mrs{ the naive composition of skeletons }reduce (↑, ↑) ◦ map (reduce (+, +)) ◦ reduce (−◦, ) ◦ rects ′◦, ) ◦ map | · |)where rects ′ = map (scan (−◦, ) ◦ map | · |) ◦ (scanr (−= { repeated application of fusions (details are omitted) }π1 ◦ ([fmrs , ⊕mrs , ⊗mrs ])= { homomorphism is composition of map and reduce }π1 ◦ map fmrs ◦ reduce (⊕mrs , ⊗mrs )
−
◦

−
◦

−
◦

=

Here, π1 is a function to extract the first component of a tuple. The cost of thederived algorithm (optimized program) is O(n3 ), while the naive one is O(n6 ); wehave succeeded in deriving the efficient parallel algorithm by eliminating redundancyof the algorithm using fusions.
Now, we have become able to develop efficient parallel algorithms with skeletons.
The rest of our concern is how to implement the algorithm as an actual parallel
program.
Implementation of algorithms described with skeletons is straightforward, when
we are provided with parallel implementation of the skeletons used, which is usuallyprovided as either libraries on existing programming languages or new languages
that include skeletons as their primitive constructs. For example, we have made alibrary on C++ to provide implementation of our designed skeletons. Using thislibrary, we can implement the optimized program of mrs as follows.
1 int mrs(const matrix<int> &mat)
2 {3
dist matrix<int> dmat(&mat);4
dist matrix<tuple> *mat2 = map(F MRS, &dmat);5
tuple *res = reduce(OplusMRS, OtimesMRS, mat2);
6
return res->first;
7 }Here, dist matrix is a data structure to be manipulated by skeletons, and F MRS,
OplusMRS, and OtimesMRS are function objects that implement the function and theoperators in the derived algorithm (these are sequential programs).

7

1.2. Homomorphism-based Structured Parallel Programming
18
speedup of mrs
16
14
12
10
8
6
4
2
0
1

4

9
processors

16

Figure 1.1. Speedup of the skeleton program for the maximum rectangle sum.
Since skeletons are implemented as components of parallel programs, programs
written with the provided skeletons work as parallel programs. For example, theabove program achieves good speed up when we use multiple processors, which isshown in Figure 1.1.
Finally, we mention about alternative notation for parallel programming basedon our skeletons. We can provide alternative notation for parallel programmingwhen we have mapping from the notation to our skeletons. For example, we can use
comprehension notation [BS90,BHS+ 94,Ble96,BG96,CK00,CK01,CKLP01,LCK06,
CLJ+ 07, FRR+ 07] as an alternative, in which we can describe computation by twoparts: one is an expression (a function) to specify element-wise computation applied
to each element in a data structure, and the other is an operator to specify reduction
computation on the results of the element-wise computation. For example, we candescribe computation of a sum of squares in comprehension notation as follows.
X
ha2 | a ∈ xi
Here, x is the data structure such as arrays, the expression
a2 specifies how toPcompute new elements from elements a in x, andspecifies how to sum themup (i.e., taking a sum with the usual plus operator). Intuitively, when x is a twodimensional
L array, the relation between the notation and our skeleton is given as
follows:
hf a | a ∈ xi ⇔ reduce (⊕, ⊕) (map f x). This notation is very usefulwhen we need nested use of skeletons. For example, the program for the maximum
rectangle sum problem is clearly written with comprehension notation as follows.
X↑h ha | a ∈ ri | r ∈ rects ′ xi
Or, it can be abbreviated as follows.
X↑h r | r ∈ rects ′ xi

8

1. Introduction

Here, we assume that we are provided with the function rects ′ , since this function
is often used for programming on two-dimensional arrays.
Also, high-level optimization like that for mrs shown above can be implemented
by an automatic optimization mechanism, because such optimization can be summarized as rewriting rules on skeleton compositions. Although such rewriting rules
may have conditions on parameter functions (operators), such conditions are usually
represented by whether the parameters have certain properties or not. Therefore, an
optimization mechanism can check such conditions and apply correct optimization,
once we annotate functions (operators) with possession of properties.
We have designed and implemented comprehension notation with a mechanism
for the high-level optimization, in which we can enjoy concise notation and thebenefit of high-level optimization for free.

1.3 Contributions and Organizations of the Thesis
The body of the thesis consists of three parts. The first part (Chapters 2 and
3) addresses principles of homomorphism-based parallel skeletons for various data
structures. The second part (Chapters 4 and 5) addresses principles of optimizations
on skeleton programs. The last part (Chapter 6) addresses implementations of theskeletons and the optimizations.
In Chapter 2, we will introduce algebras for various data structures and homomorphisms on the algebras, which are the basis of our homomorphism-based
parallel skeletons. Since homomorphisms have no gap between the structure of their
computation and the structure of their processing data, they have good composability with each other and good optimizability for their compositions. In the thesis,
we will deal with the following data structures: lists, tow-dimensional arrays, andtrees. Especially, we will propose a novel use of the abide-tree representation of
two-dimensional arrays [Bir88], of which importance has not been fully recognizedin parallel programming community.
In Chapter 3, we will design a set of parallel skeletons based on the homomorphisms for lists, two-dimensional arrays, and trees. Structured by homomorphisms, our designed skeletons have good composability with each other owing tothat of homomorphisms, which will be shown by some example programs writtenwith skeletons. Especially, we will tackle the following two challenging problem: thehomomorphism-based design of parallel skeletons on two-dimensional arrays, anddevelopment of parallel programs with these skeletons, which includes some parallel
matrix operations.
In Chapter 4, we will proceed to theories of optimizations of skeleton programs.
Parallel programs written with skeletons often suffers from the inefficiency problem caused by redundant generation of intermediate data structures among skeleton compositions. This problem is essential in skeletal parallel programming, because it supports the compositional-style programming; the skeletons are designed

1.3. Contributions and Organizations of the Thesis

9

so that we can describe many programs by composing a small set of skeletons.
Therefore, we will study fusion optimizations of skeletons based on the fusibilityof homomorphisms, which can remove intermediate data structures by fusing consecutive skeletons. Also, we will demonstrate the power of fusion optimization by
a derivation of a non-trivial efficient algorithm for the maximum rectangle sum
problem [Ben84a,Ben84b,Tak02,HIT97], giving a strategy to develop efficient skeletons programs using the fusion optimization by hand. Especially, the developmentof fusion optimization for skeletons on two-dimensional arrays and the derivationof efficient programs for two-dimensional arrays are both big contributions of thethesis.
In Chapter 5, we will study domain-specific optimizations of skeleton programs,
to solve some problems of domain-independent fusion optimizations caused by theirgenerality. We will concentrate on domain-specific fusion optimization of skeleton
programs for computation involving neighbor elements, which can be categorized into
two types: that involving a finite number of neighbor elements, such as filtering ofsequences and images, the finite difference method, and some matrix-vector operations; and that involving an infinite number of neighbor elements, such as queries ofinteresting segments on lists and rectangles (sub-arrays) on two-dimensional arrays.
We will develop domain-specific fusion rules for the former type, with proposing
a new strategy for developing domain-specific fusion optimization of skeleton programs. Then, for the latter type, we will formalize it as as nested reductions, anddevelop various theorems to provide efficient algorithms to nested reductions byfusion. Especially, characterization of domain-specific fusion rules, and optimization theorems for nested reductions with filtering or two-dimensional arrays are new
results of the thesis.
In Chapter 6, we will report implementations of the designed skeletons and the
optimizations over skeleton programs. Especially, we will propose efficient implementation of skeletons for two-dimensional arrays, a small system for fusion optimizations, and design and implementation of libraries with optimization capabilities
for structured parallel programs in Fortress [ACH+ 08]. Experimental results withthese implementation show that the developed theories are actually effective forparallel programming, and guarantee the success of the proposed skeletal programming framework.

Chapter 2
Algebras and Homomorphisms for
Data Structures
In this chapter, we will introduce algebras and their homomorphisms, which willbe the basis of homomorphism-based skeletal parallel programming. Homomorphisms are recursive functions of specific forms; their computation structures andtheir processing data structures are closely related to each other. This closenessbrings good composability of homomorphisms, and good algebraic laws for optimizing the compositions, which has been shown in the theory of Constructive Algorithmics [Bir88, Ski94, BdM96]. Therefore, using the algebras and homomorphisms as
the basis of the design, we will be able to design well-composable, well-optimizableskeletons, which will be seen in the next chapter.
First, we will introduce notation, operators, and their properties used throughout the thesis. Then, we will introduce algebras and their homomorphisms forlists [Bir88, Ski94, BdM96], two-dimensional arrays [Bir88, Mil94], and trees [Ski96,
Mat07].

2.1

Preliminaries

Notation
We will use the following notation throughout the thesis, unless otherwise noted.
Notation in the thesis follows that of Haskell [Jon02, Bir98], a pure functional
language that can describe both algorithms and algorithmic transformation concisely.
Function application is denoted by a space and the argument may be written
without brackets. Thus, f a means f (a) in ordinary notation.
Functions are curried; a function takes one argument and returns a function ora value.
The function application associates to the left. Thus, f a b means (f a) b.

12

2. Algebras and Homomorphisms for Data Structures

The function application binds more strongly than any other operator. Thus,f a ⊗ b means (f a) ⊗ b but not f (a ⊗ b).
Function composition is denoted by ◦, so (f ◦ g) x = f (g x) from its definition.
Binary operators can be used as functions by sectioning as follows: a ⊕ b =(a⊕) b = (⊕b) a = (⊕) a b.
˜ applies the operator after swapping the
For any binary operator ⊗, operator ⊗˜ b = b ⊗ a.
arguments. Thus, a ⊗For any binary operator ⊕, its identity is denoted by ı⊕ .
Pairs are Cartesian products of plural data, written like (x, y).
A projection function π1 extracts the first component of a pair. Thus, π1 (x, y) =x. These can be extended to the case of arbitrary number of elements.
Useful Binary Operators and Properties of Operators
We will introduce some useful binary operators used in the thesis. After that, we
will define some properties on operators.
The maximum and minimum operators are denoted by arrows ↑ and ↓, respectively, as follows.
a↑b =
=
a↓b =
=

a
b
a
b

if a ≥ botherwise
if a ≤ botherwise

Given a function f , the maximum and minimum operation with respect to thefunction are denoted by ↑f and ↓f , respectively, as defined below.
a ↑f b ==a ↓f b =
=

a
b
a
b

if f a ≥ f botherwise
if f a ≤ f botherwise

Given two functions f and g, binary operator × makes function f × g thatapplies the given functions respectively to the components of the argument pair(x, y). Thus, (f × g) (x, y) = (f x, g y).
Given two functions f and g, binary operator △ makes function f △ g thatapplies the given functions separately to an element, and returns the pair of theresults. Thus, (f △ g) a = (f a, g a).
Two binary operators ≪ and ≫ are defined by a ≪ b = a and a ≫ b = b,
respectively.
Now, we will define some properties of operators, which have certain importancein developing parallel programs.
Associativity is the basis of balanced parallel computation since it guarantees
correctness of re-balancing of computation tasks.

13

2.1. Preliminaries

Definition 2.1 (Associativity). Binary operator ⊕ is said to be associative if thefollowing equation holds for all a, b and c.
a ⊕ (b ⊕ c) = (a ⊕ b) ⊕ cDistributivity plays an important role in deriving efficient implementations. Basically, distributivity guarantees efficient reuse of partial results.
Definition 2.2 (Left-distributivity). Binary operator ⊗ is said to be left-distributive
over ⊕ if the following equation holds for all a, b and c.
a ⊗ (b ⊕ c) = (a ⊗ b) ⊕ (a ⊗ c)
Definition 2.3 (Right-distributivity). Binary operator ⊗ is said to be rightdistributive over ⊕ if the following equation holds for all a, b and c.
(a ⊕ b) ⊗ c = (a ⊗ c) ⊕ (b ⊗ c)
Definition 2.4 (Distributivity). Binary operator ⊗ is said to be distributive over
⊕ if ⊗ is left- and right-distributive over ⊕, i.e. the following equations hold for alla, b and c.
a ⊗ (b ⊕ c) = (a ⊗ b) ⊕ (a ⊗ c)(a ⊕ b) ⊗ c = (a ⊗ c) ⊕ (b ⊗ c)Commutativity is also used to change the order of computation to achieve goodefficiency.
Definition 2.5 (Commutativity). Binary operator ⊕ is said to be commutative if
the following equation holds for all a and b.
a⊕b=b⊕aFor example, well-known binary operators + (plus), × (times), ↑ (maximum),and ↓ (minimum) are all associative and commutative. Moreover, × distributes over+, and + distributes over ↑ or ↓. As seen later, concatenation operator ++ of lists is
associative, but not commutative. Also, function map f is left-distributive over ++,since map f (x ++ y) = (map f x) ++ (map f y). Distributivity of map f is often usedin the following derivations of efficient implementations.

14

2. Algebras and Homomorphisms for Data Structures

2.2

Base Theory of Lists

In this section, we will introduce the base theory of lists. We will start at defining
algebra of lists. Then, we will introduce homomorphism on the algebra as basic
parallel computation, in which no gap exists between the structure of computationand the structure of data. This close relationship results in good composability andgood optimizability.

2.2.1

Algebra of Lists

To represent lists, i.e., sequences of elements, we will use the following algebra oflists [Bir88].
Definition 2.6 (List algebra1 ). The algebra of lists is defined with two constructors:++ (concatenation) and [·] (singleton).
data List α = (List α) ++ (List α)| [·] αHere, [·] a, or abbreviated as [a], represents a singleton list of element a; and forany lists x and y, x ++ y represents the concatenated list consisting of elements of x
followed by those of y.
For example, a list of three elements a1 , a2 , and a3 is represented by [a1 ] ++[a2 ] ++ [a3 ], and may be abbreviated as [a1 , a2 , a3 ]. Similarly, for a list of arbitrary
number of elements a1 , a2 , . . . , an , we will use the intuitive notation [a1 , a2 , . . . , an ].
We also use a notation a : x = [a] ++ x to show the head of a list.
It is worth noting that the concatenation ++ has the associativity; for any lists
x, y, and z, two concatenations x ++ (y ++ z) and (x ++ y) ++ z result in the samelist.
The associativity of the concatenation ++ is the most important property of thealgebra, since it is the basis of good parallelism of parallel skeletons. This will be
explained after introducing the homomorphism.
Here, we will introduce a special value [ ] to represent an empty list, which issometimes useful to be used as the default value or the initial value in auxiliary
functions, although we consider computations on non-empty lists only. We assume
that the application of a function to [ ] results in [ ] when the function returns a
list, and also assume that it results in the identity of the binary operator when thefunction returns a value computed with the binary operator, which is the natural
assumption since [ ] is the identity of ++; for any list x, either of x ++ [ ] and [ ] ++ x
equals x.
1

Actually, there are several alternative algebras to represent lists [Bir88]. We have chosen thealgebra because it has good properties to be used as the basis of parallel skeletons.

15

2.2. Base Theory of Lists

2.2.2

List Homomorphism

From the theory of Constructive Algorithmics [BdM96], it follows that each constructively built-up data structure (i.e., algebraic data structure) is equipped witha powerful computation pattern called homomorphism. The homomorphism of thelist algebra is defined as follows.
Definition 2.7 (List homomorphism). Given a function f and an associative binary
operator ⊕, a list homomorphism h is a function defined by the following equations.
h (x ++ y) = h x ⊕ h y
h [a]
= f a
For notational convenience, we write ([f, ⊕]) to denote h. When it is clear from thecontext, we just call ([f, ⊕]) homomorphism.
Intuitively, a homomorphism ([f, ⊕]) is a function to replace the constructors [·]and ++ of the input list with f and ⊕, respectively. For example, applying ([f, ⊕])to a list [a1 , a2 , a3 , a4 ], we get the following result.
([f, ⊕]) [a1 , a2 , a3 , a4 ] = ([f, ⊕]) ([·] a1 ++ [·] a2 ++ [·] a3 ++ [·] a4 )= f a1 ⊕ f a2 ⊕ f a3 ⊕ f a4
Many programs are written in terms of homomorphisms. For example, a summation of elements of a list,sum [1, 2, 3, 4] = 1 + 2 + 3 + 4,
is clearly written with a homomorphism as follows:
sum = ([id , +]) .
It applies the identity function id to each of the elements, and then sums up themby the operator +:([id , +]) [1, 2, 3, 4] = id 1 + id 2 + id 3 + id 4
=1+2+3+4= sum [1, 2, 3, 4] .
Functions to calculate the length of a list and to double elements of a list are also
written as homomorphism:
length = ([one, +]) where one a = 1 ,double = ([[·] ◦ dbl, ++]) where dbl a = 2 × a .
Note that ⊕ of homomorphism ([f, ⊕]) must be associative, inheriting the associativity of ++. Otherwise the result is not well-defined, because the recursion
structure of the computation of the homomorphism can vary according to the freedom of division (construction) of the argument list.

16

2. Algebras and Homomorphisms for Data Structures

Balanced Divide-and-conquer Parallel Computation of Homomorphism
We will briefly see the good parallelism of the list algebra and the list homomorphism.
The associativity of the algebra brings good divide-and-conquer parallelism incomputation of homomorphism. Let’s consider to apply homomorphism ([f, ⊕]) tolist [a1 , a2 , . . . , an ]. To compute ([f, ⊕]) [a1 , a2 , . . . , an ] we can divide the list at the
middle point like [a1 , a2 , . . . , an ] = [a1 , a2 , . . . , a⌈n/2⌉ ] ++ [a⌈n/2⌉+1 , . . . , an ], since theconcatenation is associative. This division results in the following computation.
([f, ⊕]) [a1 , a2 , . . . , an ]= { Diving the list at the middle point }([f, ⊕]) ([a1 , a2 , . . . , a⌈n/2⌉ ] ++ [a⌈n/2⌉+1 , . . . , an ])= { Definition of homomorphism }(([f, ⊕]) [a1 , a2 , . . . , a⌈n/2⌉ ]) ⊕ (([f, ⊕]) [a⌈n/2⌉+1 , . . . , an ])The above equation means that we can compute ([f, ⊕]) [a1 , a2 , . . . , an ] in two steps:(1) computing two sub-results ([f, ⊕]) [a1 , a2 , . . . , a⌈n/2⌉ ] and ([f, ⊕]) [a⌈n/2⌉+1 , . . . , an ],and then (2) combining those results with the operator ⊕. Also, we can computethe sub-results in parallel, since their computations are independent. Thus, repeatedly applying the division of computation to get the sub-results, we can computethe result of homomorphism by a naive divide-and-conquer parallel computation.
Therefore, we can use homomorphism as the basic parallel computation pattern.

2.3

Base Theory of Two-Dimensional Arrays

We will build the base theory of two-dimensional arrays based on Constructive
Algorithmics [Bir88, Ski94, BdM96]. First, we will introduce an algebra of twodimensional arrays. Then, we will define its homomorphism, which is the basis ofparallel computations on two-dimensional arrays.

2.3.1

Abide-tree Algebra for Two-Dimensional Arrays−◦

To represent two-dimensional arrays, we use the following abide-tree algebra, which
are built up by three constructors | · | (singleton), −◦ (above) and (beside) [Bir88].
The abide-tree representation of two-dimensional arrays not only inherits the advantages of tree representations of matrices where recursive blocked algorithms canbe defined to achieve better performance [EGJK04,FW03,Wis99], but also supports
systematic development of parallel programs and architecture independent implementation.
Definition 2.8 (Abide-tree algebra). The abide-tree algebra for two-dimensional

172.3. Base Theory of Two-Dimensional Arrays(beside).

−◦arrays is defined with three constructors: | · | (singleton), −◦ (above), and−◦data AbideTree α = | · | α| (AbideTree α)−◦ (AbideTree α)| (AbideTree α) (AbideTree α)

−
◦

−
◦

Here, | · | a, or abbreviated as |a|, means a singleton array of a, i.e. a twodimensional array with a single element a. We define the function the to extract theelement from a singleton array, i.e. the |a| = a. For two-dimensional arrays x and yof the same width, x−◦ y represents an array made by putting x on y. Similarly, for
two-dimensional arrays x and y of the same height, x y represents an array madeby putting x on the left of y. Moreover, −◦ and are associative binary operatorsand have the following abide (a coined term from above and beside) property.
Definition 2.9 (Abide Property). Two binary operators ⊕ and ⊗ are said to havethe abide property or to be abiding, if the following equation holds for any x, y, u,
and y in the interested domain:(x ⊗ u) ⊕ (y ⊗ v) = (x ⊕ y) ⊗ (u ⊕ v) .

−
◦

−
◦

In the rest of the thesis, we will assume no inconsistency in height or width whenand −◦ are used.
Note that one two-dimensional array may be represented by more than one abidetrees, but these abide-trees are equivalent because of the abide property of −◦ and .
For example, we can express the 2 × 2 two-dimensional array


1 2
3 4
by the following two equivalent abide-trees.
−
◦

−
◦

−
◦

◦ (|3| |4|)(|1| |2|)−(|1|−◦ |3|) (|2|−◦ |4|)
This is in sharp contrast to other representations of matrices, such as quadtree representation [FW03], which do not allow such freedom. This freedom is important in
our framework. First, this freedom allows easy re-balancing of the tree in computations such as divide-and-conquer computations on abide-trees, as associativity of++ does for computation on lists. Therefore, the freedom is important for efficientbalanced parallel computation. Next, the freedom is useful to describe and derive
efficient programs for various architectures. In our framework, a program construction may have two phases. First, we make a general program that is architecture

18

2. Algebras and Homomorphisms for Data Structures

independent. Then, we derive a good program that may be architecture dependentor may have restricted order of access. In this approach, restrictive representation
prevents us form describing and deriving efficient programs. For example, a listof lists representation, which restricts the access order to outer dimension to innerdimension, does not allow us to describe and to derive efficient blocked algorithms.
Thus, we start with a program using the abide-tree representation that does not
impose restrictions on the access order, then we transform it to a good programthat may have the restricted order of accesses.

2.3.2

Abide-tree Homomorphism

Next, we will introduce homomorphism on the abide-trees, which will be the basisof parallel computations on two-dimensional data structures. From the theory ofConstructive Algorithmics [BdM96], it follows that each constructively built-up data
structure (i.e., algebraic data structure) is equipped with a powerful computation
pattern called homomorphism.
Definition 2.10 (Abide-tree homomorphism). Given a function f and two associative binary operators ⊕ and ⊗ that have the abide property, an abide-tree homomorphism h is a function defined by the following equations.

−
◦

h |a|
= f a
h (x−
◦ y) = h x ⊕ h yh (x y) = h x ⊗ h y
For notational convenience, we write ([f, ⊕, ⊗]) to denote h. When it is clear fromthe context, we just call ([f, ⊕, ⊗]) homomorphism.

−
◦

−
◦

Intuitively, a homomorphism ([f, ⊕, ⊗]) is a function to replace the constructors| · |, −◦, and in an input abide-tree by f , ⊕, and ⊗, respectively. We will see
in Section 3.2 that many algorithms on two-dimensional arrays can be concisely
specified by homomorphisms.
Note that ⊕ and ⊗ of homomorphism ([f, ⊕, ⊗]) must be associative and havethe abide property, inheriting the properties of −◦ and .
We can describe many functions with homomorphism. For example, a summation of elements of an array,
sum



1 5
7 3



= 1 + 5 + 7 + 3,

is clearly written with a homomorphism as follows:
sum = ([id , +, +]) .

19

2.4. Base Theory of Trees

It applies the identity function id to each of the elements, and then sums up themby the operator +:




1 5
= ([id , +, +]) 1 5 + ([id , +, +]) 7 3([id , +, +])7 3= (id 1 + id 5) + (id 7 + id 3)=1+5+7+3


1 5
.
= sum
7 3

−
◦

−
◦

−
◦

In the above computation, we assumed that the input array is constructed as (|1||5|) −◦ (|7| |3|). It is worth noting that we can get the same result when the array◦ |3|), because the plus
is constructed in the alternative way as (|1| −◦ |7|) (|5| −operator + has the abide property for itself. Generally, a commutative, associative
binary operator satisfies the condition of the abide property for itself. Functions to
calculate the width of an array and to double elements of an array are also written
as homomorphism:−◦

width = ([one, ≪, +]) where one a = 1 ,double = ([[·] ◦ dbl,−◦, ]) where dbl a = 2 × a .
Because of the flexibility of the abide-tree representation, a homomorphism([f, ⊕, ⊗]) can be implemented efficiently in parallel, which will be shown in Chapter 6. Therefore, we will design our parallel skeletons based on the homomorphisms.

2.4

Base Theory of Trees

This section briefly reviews the base theory of trees. Here, we will deal with only
binary trees, in which internal nodes have exactly two children. The theory of
another type of trees, i.e., rose trees (a term coined by Meertens [Mee88]), of whichinternal nodes have an arbitrary number of children, can be found in Matsuzaki’s
thesis [Mat07].
First, we will introduce an algebra of binary trees. Then, we will proceed to treehomomorphism.

2.4.1

Algebra of Trees

The algebra of binary trees is given as follows.
Definition 2.11 (Algebra of binary trees). The algebra of binary trees is definedwith two constructors BLeaf for leaves and BNode for internal nodes as follows.
data BTree α β = BLeaf αBNode (BTree α β) β (BTree α β)20

2. Algebras and Homomorphisms for Data Structures

The BLeaf is the constructor for leaves, and holds an element of type α. The
BNode is the constructor for internal nodes and takes three parameters, the left
subtree, the value of the node, and the right subtree, in this order. The value of thenode has type β.
Although the above algebra has no freedom for parallelism, we can constructanother equivalent algebra with certain freedom for parallelism, which has beenformalized as tree associativity [Mat07]. However, the algebra and tree associativity
are too complicated to show in this thesis. Thus, we will use the above simple
algebra to define tree skeletons, although we will show condition for efficient parallel
computation translated from the tree associativity.
We introduce two functions for manipulating binary trees. Function root b returns
the value of the root node, and function setroot b takes a binary tree and a value,
and replaces the value of the root node with the input value. Note that we use −to denote a don’t-care value.
root b (BLeaf a)root b (BNode − b −)

= a
= b

setroot b (BLeaf −) a′ = BLeaf a′setroot b (BNode l − r) b′ = BNode l b′ r

It is worth noting that there has been research on parallel skeletons for rosetrees (a term coined by Meertens [Mee88]) of which internal nodes have an arbitrary number of children. However, we will omit the results here, because it has
been developed based on the theory of binary trees. Please refer to Matsuzaki’s
thesis [Mat07] for the details.

2.4.2

Tree Homomorphisms

For binary trees, we can define the following binary-tree homomorphism (or treehomomorphism for short) [Ski94, Ski96].
Definition 2.12 (Tree Homomorphism). Let kl and kn be given functions. A function h is called tree homomorphism (or simply homomorphism), if it is defined in
the following recursive form.
h (BLeaf a)= kl ah (BNode l b r) = kn (h l) b (h r)
We may denote the tree homomorphism above as h = ([kl , kn ])b .
We can specify many tree manipulations in the form of tree homomorphism. An
example of tree homomorphism is function height b that computes the height of abinary tree.
height b (BLeaf a)= 1height b (BNode l b r) = 1 + (height b l ↑ height b r)

21

2.4. Base Theory of Trees

This function is indeed a tree homomorphism height b = ([height l , height n ])b with thetwo parameter functions defined as follows:
height l a= 1height n l b r = 1 + (l ↑ r) .
Tree homomorphisms that return a basic value, like the height b function, are often
called tree reductions.
The following two computational patterns called tree accumulations return trees
instead of basic values. These two tree accumulations are in fact tree homomorphisms [Mat07] as stated later.
Definition 2.13 (Upwards accumulation). Let kl and kn be given functions. A
function hu is called upwards accumulation, if it is defined in the following recursiveform.
hu (BLeaf a)= BLeaf (kl a)hu (BNode b l r) = let l′ = hu lr ′ = hu r
in BNode (kn b (root b l′ ) (root b r′ )) l′ r′Definition 2.14 (Downwards accumulation). Let gl and gr be given functions.
A function hd is called downwards accumulation, if it is defined in the following
recursive form with additional parameter c.
hd c (BLeaf a)= BLeaf c
hd c (BNode l b r) = BNode (hd (gl c b) l) c (hd (gr c b) r)

These tree accumulations are in fact tree homomorphisms. The upwards accumulation hu defined with two parameter functions kl and kn is a tree homomorphism,
hu = ([kl′ , kn′ ])b , in which the two parameter functions are defined as follows.
kl′ a= BLeaf (kl a)′
kn l b r = BNode l (kn b (root b l) (root b r)) r
The downwards accumulation hd defined with two functions gl and gr is a higherorder tree homomorphism, hd c t = ([kl , kn ])b t c, in which the two parameter functions are defined as follows.
kl a= λc.BLeaf c
kn b l r = λc.BNode c (l (gl c b)) (r (gr c b))

22

2.5

2. Algebras and Homomorphisms for Data Structures

Related WorkWe will briefly mention related work on representations of data structures.

List Representations
Besides the join-list representation used in this thesis, cons-list and snoc-list presentations [Bir88, BdM96] are often used in theoretical studies in the field of functional
programming. The cons-list representation has two constructors: the empty list,
and the cons operator to put an element on the head of the given list. The snoc-list
has similar constructors: the empty list, and the snoc operator to put an elementon the last of the given list. Since those two representations have no freedom forbalancing the computation of their homomorphisms, they are suitable for sequential
programs but not for parallel programs.
Fortunately, there has been research to connect the sequential representations(cons-list) with the parallel representation (join-list). The diffusion theorem [HTI99]
gives a powerful method to obtain suitable program on the join-list representationfrom a program recursively defined on the cons-list representation; the theorem
gives us a method to parallelize sequential programs. Chin et al. [HTC98, CTH98]
have studied a systematic method to derive an associative operator that plays animportant role in parallelization, based on which Xu et al. [XKH04] build an automatic derivation system for parallelizing recursive linear functions with normalization rules.

Matrix Representations
Wise et al. [Wis84] proposed representation of two-dimensional arrays by quadtrees,in which two-dimensional arrays are recursively constructed by four small sub-arraysof the same size. This representation is suitable for describing recursive blocked algorithms [EGJK04], which can provide better performance than existing algorithms
for some matrix computations such as LU and QR factorizations [FW03, Wis99].
However, the quadtree representation requires the size of two-dimensional arrays tobe the power of two. Moreover, once a two-dimensional array is represented by aquadtree, we cannot reblock the array by restructuring the quadtree, which would
prevent us from developing more parallelism in the recursive blocked algorithms onthem.
Bikshandi et al. [BGH+ 06] proposed representation of a two-dimensional arrayby a hierarchically tiled array (HTA). An HTA is an array partitioned into tiles, andthese tiles can be either conventional arrays or lower level HTAs. The outermost
tiles are distributed across processors for parallelism and the inner tiles are utilized
for locality. In the HTA programming, users are allowed to use recursive index
accessing according to the structure of HTAs, so that they can easily transformconventional programs onto HTA programs. Communication and synchronization

2.5. Related Work23

are explicitly expressed by index accessing to remote tiles. Thus HTA programs cancontrol relatively low-level parallelism and can be efficient implementation. However,
it is not presented how to derive efficient HTA programs. We think it is good that
we derive an efficient algorithm on the abide-tree then implement it on HTAs.
A more natural representation of a two-dimensional array is to use nested onedimensional arrays (lists) [Bir88, Ski94, Jeu93]. The advantage is that many results
developed for lists can be reused. However, this representation imposes much restriction on the access order of elements.
The abide-tree representation, as used in this chapter, was first proposed byBird [Bir88], as an extension of one-dimensional join lists. However, the focus there
is on derivation of sequential programs for manipulating two-dimensional arrays,
and there is little study on the framework for developing efficient parallel programs.
Our work provides a good complement.

Tree Representations
There has been research on tree representations with good parallelism for loadbalancing. Matsuzaki [Mat07] has proposed the ternary-tree representation for binary trees, in which three special nodes are introduced for the flexibility of structures.
The flexibility has been formalized as tree-associativity, which is a natural extensionof the usual associativity of binary operators.

Chapter 3
Homomorphism-based Design ofParallel Skeletons
In this chapter, we will introduce homomorphism-based design of parallel skeletonsfor various data structures. Homomorphism-based skeletons have good composability inherited from homomorphisms. Thus, we can describe many problems bycomposing a small set of homomorphism-based skeletons. Also, we will be able todevelop optimizations of skeleton compositions, owing to the nice algebraic laws ofhomomorphisms, which will be seen in the next chapter.
First, we will introduce parallel skeletons for lists. Then, we will design parallelskeletons for tow-dimensional arrays, which has remained as a challenging problem.
We will also show some non-trivial programs written with the skeletons. Finally, we
will show parallel skeletons for trees briefly.

3.1

Homomorphism-based Parallel Skeletons for
Lists

First, we will define five parallel skeletons based on lists. Then, we will show some
programs written with the skeletons, and some variants of the skeletons.

3.1.1

Definitions of Parallel Skeletons

We will define five parallel skeletons on lists: map and reduce for basic computations, zipwith for extended element-wise computations, and scan and scanr on lists
for accumulation computations. Those functions will be defined based on the homomorphism. In the theory of Constructive Algorithmics [Bir88, Ski94, BdM96],
these functions are known to be the most fundamental computation components
for manipulating algebraic data structures and for being glued together to express
complicated computations. Intuitive definitions of the skeletons are shown in Figure
3.1.

26

3. Homomorphism-based Design of Parallel Skeletons

map f [a1 , a2 , . . . , an ]reduce (⊕) [a1 , a2 , . . . , an ]zipwith f [a1 , a2 , . . . , an ] [b1 , b2 , . . . , bn ]scan (⊕) [a1 , a2 , . . . , an ]scanr (⊕) [a1 , a2 , . . . , an ]

=
=
=
=

[f a1 , f a2 , . . . , f an ]a1 ⊕ a 2 ⊕ · · · ⊕ an
[f a1 b1 , f a2 b2 , . . . , f an bn ][y1 , y2 , . . . , yn ]where yi = a1 ⊕ a2 ⊕ · · · ⊕ ai
= [z1 , z2 , . . . , zn ]where zi = ai ⊕ ai+1 ⊕ · · · ⊕ an

Figure 3.1. Intuitive definitions of parallel skeletons on lists.

Basic Skeletons: Map and Reduce
The skeletons map and reduce are two special cases of homomorphism.
The skeleton map applies the given function to each element of the given list. It
is defined as follows.
map f (x ++ y) = (map f x) ++ (map f y)
map f [a]= [f a]This definition means map f = ([[·] ◦ f, ++]).
The skeleton reduce takes a sum of the given list with the given operator. It isdefined as follows.
reduce (⊕) (x ++ y) = (reduce (⊕) x) ⊕ (reduce (⊕) y)
reduce (⊕) [a]= aThis definition means reduce (⊕) = ([id, ⊕]).
Here are example uses of the skeletons. We can use map to increment all elementsof a list as follows.
map (1+) [6, 2, 1, 4, 3, 5] = [7, 3, 2, 5, 4, 6]
We can use reduce to take a sum of a list as follows.
reduce (+) [6, 2, 1, 4, 3, 5] = 21

Extended Skeletons: Zipwith, Scan, and Scanr
The two skeletons defined above are primitive skeletons. We will define other skeletons that are extensions of these primitive skeletons.
The skeleton zipwith, an extension of map, takes two lists of the same length, andapplies the given function f to every pair of corresponding elements of the lists.
zipwith f (x ++ y) (u ++ v) = (zipwith f x u) ++ (zipwith f y v)
zipwith f [a] [b]= [f a b]

3.1. Homomorphism-based Parallel Skeletons for Lists

27

Note that in the above definition the two argument lists should be divided in theway that the corresponding sub-lists have the same length.
Here, we define a specialization of zipwith to make a list of pairs.
zip = zipwith (λx y.(x, y))We may define similar zip and zipwith for the case when the number of input lists isthree or more, and those that take k lists are denoted by zipk and zipwithk .
The skeleton scan, an extension of reduce, holds all values generated in reducing
a list by reduce.
scan (⊕) (x ++ y) = (scan (⊕) x) ⊕′ (scan (⊕) y)where sx ⊕′ sy = sx ++ map ((reduce (≫) sx) ⊕ ) sy
scan (⊕) [a]= [a]This definition means scan (⊕) = ([[·], ⊕′ ]).
Also, the skeleton scanr is defined as a reverse of scan, which holds all values
generated during a reduction in the reverse order.
scanr (⊕) (x ++ y) = (scanr (⊕) x) ⊕′′ (scanr (⊕) y)where sx ⊕′′ sy = map ( ⊕ (reduce (≪) sy)) sx ++ sy
scanr (⊕) [a]= [a]This definition means scanr (⊕) = ([[·], ⊕′′ ]).
Here are example uses of the skeletons. We can use zipwith to add two lists
element-wisely.
zipwith (+) [6, 2, 1, 4, 3, 5] [2, 7, 6, 1, 0, 4] = [8, 9, 7, 5, 3, 9]
We can use scan and scanr to compute prefix sums and suffix sums, respectively.
scan (+) [6, 2, 1, 4, 3, 5] = [6, 8, 9, 13, 16, 21]
scanr (+) [6, 2, 1, 4, 3, 5] = [21, 15, 13, 12, 8, 5]

3.1.2

Example Programs and Variants of Skeletons

Composing these skeletons defined above, we can describe many useful functions asfollows.
idreverseflattenlength
last
head

=
=
=
=
=
=

reduce (++) ◦ map [·]reduce (+̃+) ◦ map [·]reduce (++)
reduce (+) ◦ map (λx. 1)
reduce (≫)
reduce (≪)

28

3. Homomorphism-based Design of Parallel Skeletons

The function id is the identity function of lists. The function reverse reverses thegiven list. The function flatten flattens a list of lists into a list. The function length
returns the length of the given list. The functions last and head returns the lastelement and the first element of the given list, respectively.
Finally, we will defined some variants of skeletons and useful functions definedwith them.
scan′ (⊕) e = map (e⊕) ◦ scan (⊕)
scanr′ (⊕) e = map (⊕e) ◦ scanr (⊕)
shift≫ e= map π1 ◦ scan′ (⊕) ( , e, ) ◦ map f
where f a = (a, a, True)( , p, ) ⊕ ( , c, True) = (p, c, False)( , , ) ⊕ (p, c, False) = (p, c, False)
shift≪ e= map π1 ◦ scanr′ (⊕) ( , e, ) ◦ map f
where f a = (a, a, True)( , c, True) ⊕ ( , p, ) = (p, c, False)(p, c, False) ⊕ ( , , ) = (p, c, False)filter p= ([if p a then [a] else [ ], ++])
init
= flatten ◦ shift≫ [ ] ◦ map [·]tail= flatten ◦ shift≪ [ ] ◦ map [·]drop n x= tail (drop (n − 1) x)
drop 0 x= x
take n x= [head x] ++ take (n − 1) (tail x)
take 0 x= []taker n= reverse ◦ take n ◦ reverse
dropr n= reverse ◦ drop n ◦ reverse

Their intuitive definition is shown below.
scan′ (⊕) e [a1 , . . . , an ]= [b1 , . . . , bn ]where bi = e ⊕ a1 ⊕ · · · ⊕ ai
scanr′ (⊕) e [a1 , . . . , an ] = [c1 , . . . , cn ]where ci = ai ⊕ · · · ⊕ an ⊕ eshift≫ e [a1 , . . . , an ]= [e, a1 , . . . , an−1 ]shift≪ e [a1 , . . . , an ]= [a2 , . . . , an , e]filter p [a1 , a2 , . . . , an ]= [ai1 , ai2 , . . . , aim ] where p (aij ) = True
init [a1 , . . . , an ]= [a1 , . . . , an−1 ]tail [a1 , . . . , an ]= [a2 , . . . , an ]take k [a1 , . . . , an ]= [a1 , . . . , ak ]drop k [a1 , . . . , an ]= [ak+1 , . . . , an ]taker k [a1 , . . . , an ]= [an−k+1 , . . . , an ]dropr k [a1 , . . . , an ]= [a1 , . . . , an−k ]Skeletons scan′ and scanr′ take the initial values of reductions. Skeleton shift≫ shifts
the elements by putting the given element to the head and discarding the last element. Skeleton shift≪ does the converse. Two functions init and tail take initial3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays29

segment and the tail segment of the list, respectively. The function take k picks
up the first k elements of the given list, and the function drop removes the first kelements from the given list. Conversely, the function taker k picks up the last kelements of the given list, and the faction dropr removes the last k elements fromthe given list.

3.2

Homomorphism-based Parallel Skeletons for
Two-Dimensional Arrays

In this section, we will design a set of parallel skeletons for manipulating twodimensional arrays. Also, we will show that compositions of the skeletons are powerful enough to describe useful parallel algorithms. The power of skeleton compositions
will be shown by describing nontrivial problems such as matrix multiplication and
QR decomposition.

3.2.1

Definitions of Parallel Skeletons

We will define five parallel skeletons on two-dimensional arrays: map and reduce forbasic computations, zipwith for extended element-wise computations, and scan andscanrfor accumulation computations. In the theory of Constructive Algorithmics[Bir88, Ski94, BdM96], list versions of these functions are known to be the mostfundamental computation components for manipulating algebraic data structuresand for being glued together to express complicated computations.
Intuitive definitions of the skeletons are shown in Figure 3.2.

Basic Skeletons: Map and Reduce
The skeletons map and reduce are two special cases of homomorphism.
The skeleton map applies the given function f to every element of the given
two-dimensional array. Its definition is as follows.

−
◦

−
◦

map f |a|= |f a|map f (x−◦ y) = (map f x)−◦ (map f y)
map f (x y) = (map f x) (map f y)−◦

This definition means map f = ([| · | ◦ f,−◦, ]).
The skeleton reduce collapses a two-dimensional array into a value using two
associative binary operators ⊕ and ⊗ that have the abide property. In other words,
the skeleton reduce takes a sum of the given array with the given operators, in which
the sum of vertical direction is computed with the first operator ⊕, and the sum ofhorizontal direction is computed with the second operator ⊗. Its definition is given

30

3. Homomorphism-based Design of Parallel Skeletons



x11
 x21

map f  .
 ..

x12
x22
..
.


x1n
x2n 

.. 
. 

···
···
..
.

xm1 xm2 · · ·

xmn



···
···
..
.

x11
 x21

reduce (⊕, ⊗)  .
 ..

x12
x22
..
.



···
···
..
.



f x11
 f x21

=  .
 ..

x12
x22
..
.


x1n
x2n 

..  =
. 

xmn


y12
y22
..
.


y1n
y2n 

.. 
. 

···
···
..
.

ym1 ym2 · · ·ymn

f x12 y12f x22 y22
..
.

···
···
..
.

f xm1 ym1 f xm2 ym2 · · ·

x11
 x21

scan (⊕, ⊗)  .
 ..

x12
x22
..
.

···
···
..
.

xm1 xm2 · · ·


x1n
x2n 

.. 
. 

xmn

f xmn

(xm1 ⊗ xm2 ⊗ · · · ⊗ xmn )f x11 y11
 f x21 y21

=
..

.



f x1n
x 
f2n

.. 
. 

(x11 ⊗ x12 ⊗ · · · ⊗ x1n )⊕(x21 ⊗ x22 ⊗ · · · ⊗ x2n )⊕
..
.

xmn

 
x1n
y11
 y21
x2n 
 
..   ..
.   .

xm1 xm2 · · ·

···
···
..
.

f xm1 f xm2 · · ·xm1 xm2 · · ·

x11
 x21

zipwith f  .
 ..

f x12
f x22
..
.



y11
 y21

=  .
 ..

y12
y22
..
.

···
···
..
.

ym1 ym2 · · ·where yij =
f x1n y1n
f x2n y2n 


..

.

f xmn ymn

y1n
y2n 

.. 
. 

ymn

(x11 ⊗ x12 ⊗ · · · ⊗ x1j )⊕(x21 ⊗ x22 ⊗ · · · ⊗ x2j )⊕..
.
(xi1 ⊗ xi2 ⊗ · · · ⊗ xij )



x11
 x21

scanr (⊕, ⊗)  .
 ..

x12
x22
..
.

···
···
..
.

xm1 xm2 · · ·


x1n
x2n 

.. 
. 

xmn



z11
 z21

=  .
 ..

z12
z22
..
.

···
···
..
.

zm1 zm2 · · ·where zij =


z1n
z2n 

.. 
. 

zmn

(xij ⊗ · · · ⊗ xin )⊕..
.
(xmj ⊗ · · · ⊗ xmn )Figure 3.2. Intuitive definition of five skeletons on two-dimensional arrays.

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays31

as follows.

−◦

reduce (⊕, ⊗) |a|= a
reduce (⊕, ⊗) (x−◦y) = (reduce (⊕, ⊗) x)⊕(reduce (⊕, ⊗) y)
reduce (⊕, ⊗) (x y ) = (reduce (⊕, ⊗) x)⊗(reduce (⊕, ⊗) y) ,This definition means reduce(⊕, ⊗) = ([id, ⊕, ⊗]).
Here are some example uses of the skeletons. We can use map to take a scalar
product of an array as follows.

12 4 26 2 1=
map (2×)
8 6 104 3 5We can use reduce to take a sum of an array as follows.


6 2 1
= 21
reduce (+, +)4 3 5Extended Skeleton for Element-wise Computation: Zipwith
The two skeletons defined above are primitive skeletons. We define other skeletonsthat are extensions of these primitive skeletons.
The skeleton zipwith, an extension of map, takes two two-dimensional arrays ofthe same shape, and applies a function f to every pair of corresponding elements of
the arrays.

−
◦

−
◦

−
◦

zipwith f |a| |b|= |f a b|
zipwith f (x−◦ y) (u−◦ v) = (zipwith f x u)−◦ (zipwith f y v)
zipwith f (x y) (u v) = (zipwith f x u) (zipwith f y v)
Note that in the above definition the argument two-dimensional arrays of the function should be divided in the way that the sizes of x and u are the same.
We also define useful function zip, which is a specialization of zipwith to make a
two-dimensional array of pairs of corresponding elements.
zip (u, v) = zipwith (λxy. (x, y)) u v
We may define similar zip and zipwith for the case when the number of input arraysis three or more, and those that take k arrays are denoted by zipk and zipwithk .
Also, we define unzip to be the inverse of zip.
For example, we can use zipwith to compute an addition of two arrays as follows.





6 10 3
0 8 2
6 2 1
=
zipwith (+)
13 4 12
9 1 7
4 3 5

32

3. Homomorphism-based Design of Parallel Skeletons

Simple Functions Described with Map, Reduce, and Zipwith
Composing these skeletons defined above, we can describe many useful functions asfollows. We will use these functions to define complex skeletons scan and scanr.

−
◦ −
◦

−
◦

−
◦ −
◦

−
◦ −
◦

−
◦

−
◦

id
= reduce(−◦, ) ◦ map | · |tr= reduce( , −◦) ◦ map | · |˜◦, ) ◦ map | · |rev= reduce(−̃flatten= reduce(−◦, )height= reduce(+, ≪) ◦ map (λx. 1)
width= reduce(≪, +) ◦ map (λx. 1)
cols= reduce(zipwith(−◦), ) ◦ map || · ||rows
= reduce(−◦, zipwith( )) ◦ map || · ||reducec (⊕) = map(reduce(⊕, ≪)) ◦ cols
reducer (⊗) = map(reduce(≪, ⊗)) ◦ rows
mapc f= reduce(≪, ) ◦ map f ◦ cols
mapr f= reduce(−◦, ≪) ◦ map f ◦ rows
madd
= zipwith(+)
msub= zipwith(−)
top= reduce(≫, ) ◦ map | · |bottom= reduce(≫, ) ◦ map | · |left= reduce(−◦, ≫) ◦ map | · |right= reduce(−◦, ≫) ◦ map | · |Here, || · || is abbreviation of | · | ◦ | · |. The function id is the identity function of
AbideTree, and tr is the matrix-transposing function. The function rev takes a twodimensional array and returns the array reversed in the vertical and the horizontal
direction. The function flatten flattens nested arrays. The functions height andwidth return the number of rows and columns, respectively. The functions colsand rows return arrays of which elements are columns and rows of the array of
the argument, respectively. The functions reducec and reducer are specializationsof reduce to collapse two-dimensional arrays in column direction and row direction,
respectively. They return a row-vector (an array of which height is one) and acolumn-vector (an array of which width is one), respectively. The functions mapc
and mapr are specializations of map to apply functions to each column and row,
respectively (i.e. the function of the argument takes column-vector or row-vector).
The functions madd and msub denote matrix addition and subtraction, respectively.
The functions top, bottom, left, and right return the bottom row, the top row, theleftmost column, and the rightmost column, respectively.

Extended Skeleton for Accumulation Computation: Scan andScanr
We will define the rest of the extended skeletons.

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays

33

The skeleton scan, an extension of reduce, holds all values generated in reducing
a two-dimensional array by reduce. Its definition is as follows.

−◦scan (⊕, ⊗) |a|= |a|scan (⊕, ⊗) (x−◦ y) = (scan (⊕, ⊗) x) ⊕′ (scan (⊕, ⊗) y)scan (⊕, ⊗) (x y) = (scan (⊕, ⊗) x) ⊗′ (scan (⊕, ⊗) y)
Here operators ⊕′ and ⊗′ are defined as follows.

−◦sx ⊕′ sy = sx−◦ sy ′
where sy ′ = mapr (zipwith (⊕) (bottom sx)) sysx ⊗′ sy = sx sy ′where sy ′ = mapc (zipwith (⊗) (right sx)) sy
The definition means scan (⊕, ⊗) = ([| · |, ⊕′ , ⊗′ ]). Note that ⊕ and ⊗ must be
associative and have the abide property.
The skeleton scanr is the reverse of scan, i.e., holds all values generated during
the reduction in the reverse order. Its definition is as follows.

−◦

scanr (⊕, ⊗) |a|= |a|
scanr (⊕, ⊗) (x−◦ y) = (scanr (⊕, ⊗) x) ⊕′ (scanr (⊕, ⊗) y)scanr (⊕, ⊗) (x y) = (scanr (⊕, ⊗) x) ⊗′ (scanr (⊕, ⊗) y)
Here operators ⊕′ and ⊗′ are defined as follows.

−
◦

◦ sy
sx ⊕′ sy = sx′ −where sx′ = mapr (λx. zipwith (⊕) x (top sy)) sx′
sx ⊗ sy = sx′ sy
where sx′ = mapc (λx. zipwith (⊗) x (left sy)) sx
The definition means scanr (⊕, ⊗) = ([| · |, ⊕′ , ⊗′ ]). Note that ⊕ and ⊗ must be
associative and have the abide property.
Here are example uses of the skeletons. We can compute a prefix rectangle sum(an upper-left prefix sum) and a suffix rectangle sum (a lower-right suffix sum) using
scan and scanr, respectively.




6 8 9
6 2 1
=
scan (+, +) 10 15 21
4 3 5 
21 11 6
6 2 1=
scanr (+, +)12 8 54 3 5Using the skeletons scan and scanr, we can define allredr and allredc to the resultsof reducer and reducec in rows and columns, respectively. These functions are defined
as follows. We will use them in later sections.
allredc (⊕) = scanr(≫, ≪) ◦ scan(⊕, ≫)allredr (⊗) = scanr(≪, ≫) ◦ scan(≫, ⊗)34

3. Homomorphism-based Design of Parallel Skeletons

Composition of the Skeletons: Accumulate
Finally, we will define a complex general skeleton accumulate that can deal with accumulation parameters. This skeleton is sometimes useful to develop general theory
for skeletons. Also, we will show that the skeleton is a composition of the previous
skeletons.
The idea of this skeleton is that we can calculate the internal value of a subarray
if we know the intermediate result on the edges of upper and left subarrays. Thisidea is quite natural, because the cross section of a two-dimensional array is anedge, and the minimum information that should be transmitted between them isthe edge. As seen later, this skeleton is a general skeleton in the sense that it
contains all skeletons defined above.
accumulate (x1 −◦ x2 ) (e, u, v1 −◦ v 2 ) = y1 ⊕ ′ y2
wherey1 = accumulate x1 (e, u, v1 )y2 = accumulate x2 (e2 , u2 , v2 )e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map q x1 ))
−
◦

−
◦

accumulate (x1 x2 ) (e, u1 u2 , v) = y1 ⊗′ y2
where
y1 = accumulate x1 (e, u1 , v)
y2 = accumulate x2 (e2 , u2 , v2 )e2 = e ⊗ reduce (⊕, ⊗) u1
u2 = zipwith (⊗) v (reducer (⊗) (map q x1 ))accumulate |a| (e, |u|, |v|) = p a ((e ⊗ u) ⊕ (v ⊗ q a))
To create the accumulation parameters, the skeleton accumulate collapses the argument two-dimensional array with a function q and two abiding operators ⊕ and⊗, and puts them on each element. Then, to get the result value it consumes theaccumulation parameters and the element by replacing constructors with a function
p and operators ⊕′ and ⊗′ . The meaning of the accumulation parameters is asfollows: e is the resulting value of collapsing the upper-left subarray of the argument array with the operators ⊕ and ⊗, u is the resulting row-vector of collapsingeach column of the upper subarray with the operator ⊕, and v is the resulting
column-vector of collapsing each row of the left subarray with the operator ⊗ . We
write [[(⊕′ , ⊗′ , p), (⊕, ⊗, q)]] to denote this accumulate. Here, (⊕, ⊗, q) corresponds
to the computation of accumulation parameters and (⊕′ , ⊗′ , p) corresponds to thecomputation of resulting value.
The skeleton accumulate has somewhat complicated definition, however, it canbe described as a composition of the other skeletons defined above. We give here thetheorem showing the relation between accumulate and other skeletons. The functions

35

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays

dropc and dropr used in the theorem drop left-columns and upper-rows respectively:−◦

dropc n (x y) = y s.t. width x = n ,dropr n (x−◦ y) = y s.t. height x = n .

This theorem is an extension of the diffusion theorem on lists [HTI99].
Theorem 3.1 (Diffusion). The skeleton accumulate can be diffused to the composition of other skeletons as follows.

Proof. The theorem is proved by induction on the structure of abide-trees.
The base case is proved as follows.

=

=

=

−
◦

=

−
◦

=

−
◦

=

−
◦

=

LHS |a| (e, |u|, |v|)
{ The left hand side }[[(⊕′ , ⊗′ , p), (⊕, ⊗, q)]] |a| (e, |u|, |v|)
{ Definition of accumulate }p a ((e ⊗ u) ⊕ (v ⊗ q a)){ Definition of reduce }reduce (⊕′ , ⊗′ ) |p a ((e ⊗ u) ⊕ (v ⊗ q a))|{ Definition of zipwith }let x′′′′ = zipwith p |a| |((e ⊗ u) ⊕ (v ⊗ q a))|in reduce (⊕′ , ⊗′ ) x′′′′{ Definition of scan, dropr and dropc }let x′′ = scan (⊕, ⊗) ((|e| |u|)−◦ (|v| |a|))x′′′ = dropc 1 (dropr 1 x′′ )x′′′′ = zipwith p |a| |((e ⊗ u) ⊕ (v ⊗ q a))|in reduce (⊕′ , ⊗′ ) x′′′′{ Definition of map }let x′ = map q |a|x′′ = scan (⊕, ⊗) ((|e| |u|)−◦ (|v| x′ ))x′′′ = dropc 1 (dropr 1 x′′ )x′′′′ = zipwith p |a| |((e ⊗ u) ⊕ (v ⊗ q a))|in reduce (⊕′ , ⊗′ ) x′′′′{ The right hand side }RHS |a| (e, |u|, |v|)

The induction case for −◦ is proved as follows.

LHS (x1 −◦ x2 ) (e, u, v1 −◦ v2 )={ The left hand side }[[(⊕′ , ⊗′ , p), (⊕, ⊗, q)]] (x1 −◦ x2 ) (e, u, v1 −◦ v2 )

x))

−
◦

−
◦

[[(⊕′ , ⊗′ , p), (⊕, ⊗, q)]] x (e, u, v) = let x′ = map q x◦ (vx′′ = scan(⊕, ⊗) ((|e| u)−x′′′ = dropc 1 (dropr 1 x′′ )x′′′′ = zipwith p x x′′′in reduce(⊕′ , ⊗′ ) x′′′′36

3. Homomorphism-based Design of Parallel Skeletons

−
◦

−
◦

−
◦
−
◦

−
◦

◦
−
◦ −
−
◦

−
◦

{ Definition of accumulate }let e2 = e ⊕ reduce(⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map q x1 ))y1 = accumulate x1 (e, u, v1 )y2 = accumulate x2 (e2 , u2 , v2 )in y1 ⊕′ y2={ Hypothesis of induction }let e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map q x1 ))y1 = let x′1 = map q x1
x′′1 = scan (⊕, ⊗) ((|e| u)−◦ (v1 x′1 ))′′′′′x1 = dropc 1 (dropr 1 x1 )
′′′
x′′′′
1 = zipwith p x1 x1
in reduce (⊕′ , ⊗′ ) x′′′′1y2 = let x′2 = map q x2
◦ (v2 x′2 ))x′′2 = scan (⊕, ⊗) ((|e2 | u2 )−′′′′′x2 = dropc 1 (dropr 1 x2 )
′′′
x′′′′
2 = zipwith p x2 x2
in reduce (⊕′ , ⊗′ ) x′′′′2′
in y1 ⊕ y2=
{ Expansion of inner let }let e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map q x1 ))
x′1 = map q x1
x′2 = map q x2
◦ (v1 x′1 ))x′′1 = scan (⊕, ⊗) ((|e| u)−◦ (v2 x′2 ))x′′2 = scan (⊕, ⊗) ((|e2 | u2 )−′′′′′x1 = dropc 1 (dropr 1 x1 )′′x′′′
2 = dropc 1 (dropr 1 x2 )′′′′′′′x1 = zipwith p x1 x1
′′′
x′′′′
2 = zipwith p x2 x2
′
′
′
′′′′
in reduce (⊕′ , ⊗′ ) x′′′′1 ⊕ reduce (⊕ , ⊗ ) x2=
{ Property of scan and edges proved below }let x′1 = map q x1
x′2 = map q x2
◦ (v1 x′1 ))x′′1 = scan (⊕, ⊗) ((|e| u)−x′′2 = mapr (zipwith (⊕) (bottom x′′1 )) (scan (⊕, ⊗) (v2′′x′′′
1 = dropc 1 (dropr 1 x1 )′′x′′′
2 = dropc 1 x2
′′′
x′′′′
1 = zipwith p x1 x1′′′′
x2 = zipwith p x2 x′′′2′
in reduce (⊕′ , ⊗′ ) x′′′′⊕reduce (⊕′ , ⊗′ ) x′′′′
1
2
−
◦

=

−
◦

x′2 ))

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays{ Combining x1 and x2 by −◦ }◦ x′2 ) = map q (x1 −◦ x2 )let (x′1 −(x′′1 −◦ (v1 x′1 ))−◦ (v2◦ x′′2 ) = scan (⊕, ⊗) (((|e| u)−
′′′
′′
′′′
′′
◦ x2 ))◦ x2 ) = dropc 1 (dropr 1 (x1 −
(x1 −
′′′′
′′′′
◦ x′′′◦ x2 ) = zipwith p (x1 −◦ x2 ) (x′′′
(x1 −
2)
1 −
′′′′
−
in reduce (⊕′ , ⊗′ ) (x′′′′
◦
x
)
1
2
=
{ Renaming internal variables }let x′ = map q (x1 −◦ x2 )′′
x = scan (⊕, ⊗) (((|e| u)−◦ ((v1 −◦ v2 ) x′ )))x′′′ = dropc 1 (dropr 1 x′′ )x′′′′ = zipwith p (x1 −◦ x2 ) x′′′in reduce (⊕′ , ⊗′ ) x′′′′={ The right hand side }RHS (x1 −◦ x2 ) (e, u, v1 −◦ v2 )

37

−
◦

−
◦

x′2 ))

−
◦

−
◦

−
◦

=

−
◦

The induction case for is proved similarly.
To complete the above proof, we prove the following property of scan and edges.
This property says that the bottom of the result of scan to an array is equivalent
to that of scan applied to the row vector resulting by collapsing each column of thearray.
bottom (scan (⊕, ⊗) x) = scan (⊕, ⊗) (reducec (⊕) x)
The following instantiation of this property completes the above proof.
−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

◦ (v1 x′1 ))) = scan (⊕, ⊗) (|e2 | u2 )bottom (scan (⊕, ⊗) ((|e| u)−where (|e2 | u2 ) = (e ⊕ reduce (⊕, ⊗)v1 ) (zipwith (⊕) u (reducec (⊕) x′1 ))= reducec (⊕) ((|e| u)−◦ (v1 x′1 ))The property is proved by induction on the structure of abide-trees. Base caseis proved as follows.
bottom (scan (⊕, ⊗) |a|){ Definition of scan }bottom |a|={ Definition of bottom }|a|={ Definition of scan }scan (⊕, ⊗) |a|={ Definition of reducec }scan (⊕, ⊗) (reducec (⊕) |a|)=38

3. Homomorphism-based Design of Parallel Skeletonsis proved as follows.

−◦

Inductive case for

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

bottom (scan (⊕, ⊗) (x1 x2 ))= { Definition of scan }bottom (scan (⊕, ⊗) x1
mapc (zipwith(⊗) (right (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ))= { bottom distributes over }bottom (scan (⊕, ⊗) x1 )bottom (mapc (zipwith(⊗) (right (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ))= { Promote bottom through mapc }bottom (scan (⊕, ⊗) x1 )mapc (zipwith(⊗) (bottom (right (scan (⊕, ⊗) x1 )))) (bottom (scan (⊕, ⊗) x2 )))= { bottom ◦ right = right ◦ bottom }bottom (scan (⊕, ⊗) x1 )mapc (zipwith(⊗) (right (bottom (scan (⊕, ⊗) x1 )))) (bottom (scan (⊕, ⊗) x2 )))= { Hypothesis of induction }scan (⊕, ⊗) (reducec (⊕) x1 )mapc (zipwith(⊗) (right (scan (⊕, ⊗) (reducec (⊕) x1 ))))
(scan (⊕, ⊗) (reducec (⊕) x2 ))= { Definition of scan }scan (⊕, ⊗) (reducec (⊕) x1 reducec (⊕) x2 )= { Definition of reducec }scan (⊕, ⊗) (reducec (⊕) (x1 x2 ))Inductive case for −◦ is proved as follows.
bottom (scan (⊕, ⊗) (x1 −◦ x2 ))={ Definition of scan }bottom (scan (⊕, ⊗) x1−◦ mapr (zipwith(⊕) (bottom (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ))={ bottom eliminates the upper subarray of −◦ }bottom (mapr (zipwith(⊕) (bottom (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ))={ Promote bottom through mapr }mapr (zipwith(⊕) (bottom (scan (⊕, ⊗) x1 ))) (bottom (scan (⊕, ⊗) x2 ))={ Application of mapr f to a row vector is application of f to the vector }zipwith(⊕) (bottom (scan (⊕, ⊗) x1 )) (bottom (scan (⊕, ⊗) x2 ))={ Hypothesis of induction }zipwith(⊕) (scan (⊕, ⊗) (reducec (⊕) x1 )) (scan (⊕, ⊗) (reducec (⊕) x2 ))={ Promotion of zipwith(⊕) through scan (⊕, ⊗) to row vectors }scan (⊕, ⊗) (zipwith(⊕) (reducec (⊕) x1 ) (reducec (⊕) x2 ))= { Definition of reducec }scan (⊕, ⊗) (reducec (⊕) (x1 −◦ x2 ))

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays

39

This theorem says that we have an efficient parallel implementation of accumulate
if the other skeletons have efficient parallel implementations. Moreover, we can write
map, reduce, scan in accumulate as follows.

−
◦

−
◦

map f x= [[(−◦, , λa.λe.(| · | ◦ f ) a), ( , , )]] x
reduce (⊕) (⊗) x = [[(⊕, ⊗, λa.λe.a), ( , , )]] x
scan (⊕) (⊗) x= [[(−◦, , λa.λe.| · | e), (⊕, ⊗, id )]] x e
where e = (ı, [ı, . . . , ı], [ı, . . . , ı])ı ⊕ a = a, ı ⊗ a = a
Here, ‘ ’ denotes “don’t care”. This property is sometimes useful to develop general
theory for manipulating skeletons.

Relationship between Directed and Non-directed Skeletons
It should be noted that reduce can be expressed by its directed versions reducec andreducer , when two binary operators ⊕ and ⊗ are abiding.
reduce (⊕, ⊗) = the ◦ reducec (⊕) ◦ reducer (⊗)
reduce (⊕, ⊗) = the ◦ reducer (⊗) ◦ reducec (⊕)(3.2)

Like reduce, we may define scan↓ and scan→ that are specialization of scan andscan a two-dimensional array in column and row direction respectively:scan↓ (⊕) = scan (⊕, ≫)scan→ (⊗) = scan (≫, ⊗) ;scan can be expressed by scan↓ and scan→ when two binary operators ⊕ and ⊗ are
abiding.
scan (⊕, ⊗) = scan↓ (⊕) ◦ scan→ (⊗)
scan (⊕, ⊗) = scan→ (⊗) ◦ scan↓ (⊕)

3.2.2

(3.3)

Example Complex Programs Described with Parallel
Skeletons

In this section, we will show two programs written with parallel skeletons definedso far. One example is matrix multiplication, which will be fully described with theskeletons. The other example is QR factorization, which will be partially describedwith the skeletons, and the whole computation will be defined as a mutual recursivefunction. The important point here is that we do not necessarily need to describe
the whole computation with skeletons to get parallel programs.

40

3. Homomorphism-based Design of Parallel Skeletons

Matrix Multiplication
As an involved example, we will describe a parallel algorithm for matrix multiplication, which is a primitive operation of matrices, with the defined parallel skeletonson two-dimensional arrays.
The following program is an intuitively-understandable description of matrix
multiplication, in which an element of the resulting matrix is an inner product ofa row vector and a column vector of the input matrices. This program simply
implements the definition of matrix multiplication. Although users do not need toconsider parallelism at all, this program can be executed in parallel due to parallelismof each skeleton.

−◦

mm = zipwithP iprod ◦ (allrows × allcols)
whereallrows = allredr ( ) ◦ map | · |allcols = allredc (−◦) ◦ map | · |iprod = (reduce (+, +)◦) ◦ zipwith(×) ◦ tr
zipwithP (⊗) (x, y) = zipwith (⊗) x y
Although this definition seems to use O(n3 ) memory space for n × n matrices due toduplications of rows and columns with allredr and allredc , we can execute this multiplication using O(n2 ) memory space. This is because we can use references insteadof duplications in the implementation of allredr and allredc due to the properties ofthe operators (≫ and ≪) in their definitions. This kind of optimization is currently
done by hand. However, we think it will be automatically done by compilers or
libraries when they take the properties of the operators into account.
Since the program is written with skeletons, we can easily generalize the computation of matrix multiplication as follows.

−◦

gemm (⊕, ⊗) = zipwithP iprod ◦ (allrows ⊗ allcols)
whereallrows = allredr ( ) ◦ map | · |allcols = allredc (−◦) ◦ map | · |iprod = (reduce (⊕, ⊕)◦) ◦ zipwith(⊗) ◦ tr
zipwithP (⊗) (x, y) = zipwith (⊗) x y
The generalized matrix multiplication gemm (⊕, ⊗) computes the multiplication
with the given two operators ⊕ and ⊗ instead of the usual operators + and ×,
respectively. For example, the generalized matrix multiplication is useful to computethe distance matrices with the maximum operator and the plus operator; the shortest
one-hop distances between points are computed by multiplication of matrices ofwhich (i, j) element is the direct distance between point i and point j. Other
application of the generalized matrix multiplication will be found in Chapter 4.3.

3.2. Homomorphism-based Parallel Skeletons for Two-Dimensional Arrays

41

QR Factorization
As the other nontrivial example, we show descriptions of two parallel algorithms
for QR factorization [EGJK04]. Given a matrix A, QR factorization computes anorthogonal matrix Q and an upper-triangular matrix R such that A = QR. We
will not explain the details, but we hope to show that these algorithms can be dealt
with in our framework. The important point here is that the whole algorithm does
not necessarily be able to be written with skeletons, but we can write some partsof the algorithm with skeletons, and the whole computation may be written as arecursive function. In this case, we can do parallel computation in both those parts
described with skeletons and independent recursive calls in the recursive definitionof the algorithm.
We give the recursive description of a QR factorization algorithm based on Householder transform. This function returns Q and R which satisfy A = QR where A
is a matrix of m × n, Q an orthogonal matrix of m × m and R an upper-triangularmatrix of m × n.

−
◦

−
◦

−
◦

−
◦

−
◦

qr ((A11 −◦ A21 ) (A12 −◦ A22 ))
= let (Q1 , R11 −◦ O) = qr (A11 −◦ A21 )
ˆ
(R12 −◦ A22 ) = mm (tr Q1 ) (A12 −◦ A22 )
ˆ(Q̂2 , R22 ) = qr A22◦ (O Q̂2 ))Q = mm Q1 ((I O)−◦ (O R22 ))in (Q, (R11 R12 )−qr (|a|−◦ x) = hh (|a|−
◦ x)
hh v
= let v ′ = pmadd v e
a = reduce (+, +) (zipwith(×) v ′ v ′ )u = map (/a) v ′Q = msub I (map (×2) (mm u (tr u)))in (Q, e)Here e is a vector (a matrix of which width is 1) whose first element is 1 and theother elements are 0, and I and O represent an identity matrix and a zero matrix
of suitable size respectively.
Furthermore, we give the recursive description of QR factorization algorithm
on quadtree [FW03]; describing algorithms on quadtrees to those on abide-trees isalways possible because abide-trees is more flexible than quadtrees, although weneed to impose some restriction on division of the argument arrays. This function
qrq is mutual recursively defined with an extra function e, and returns Q and R thatsatisfy A = QR where A is a matrix of n × n (n = 2k for a natural number k), Q is

42

3. Homomorphism-based Design of Parallel Skeletons

an orthogonal matrix of n × n and R is an upper-triangular matrix of n × n.

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

qrq |a| = (|1|, |a|)
◦ A22 ))
qrq ((A11 −◦ A21 ) (A12 −= let (Q1 , R1 ) = qrq A11(Q2 , R2 ) = qrq A21
Q12 = (Q1 O)−◦ (O Q2 )(Q3 , R3 ) = e (R1 , R2 )Q4 = mm Q12 Q3(Un −◦ Us ) = mm (tr Q4 ) (A12 −◦ A22 )
(Q6 , R6 ) = qrq Us
Q = mm Q4 ((I O)−◦ (O Q6 ))◦ (O R6 )
R = (R3 Un )−in (Q, R)
Note that Aij (i, j ∈ {1, 2}) have the same shape. A definition of the involved extra
function e is as follows.

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦
−
◦
−
◦
−
◦

−
◦

−
◦

−
◦
−
◦

−
◦
−
◦
◦
−
◦ −

−
◦ −
◦

−
◦

◦
−
◦ −

−
◦
−
◦ −
◦
−
◦
−
◦
−
◦
−
◦
−
◦

−
◦

−
◦
−
◦
−
◦
−
◦
−
◦

−
◦

−
◦

−
◦ −
◦ −
◦

−
◦

e (N, O) = (I, N )e (|n|, |s|) = let Q = g(n, s)(N, O) = mm(tr Q) (|n|−◦ |s|)in (Q, N )e ((N11 −◦ N21 ) (N12 −◦ N22 ), (S11 −◦ S22 ))◦ S21 ) (S12 −
= let
(Q12
◦ Q22
((Q11
◦ Q21
1 −
1 ), N1 ) = e (N11 , S11 )
1 −
1 )
12
21
22
−
−
((Q11
(Q
◦
Q
◦
Q
)
2
2
2
2 ), N2 ) = e (N22 , S22 )12
11◦ (O −◦ Q11
O Q12
Q12 = (Q1 O Q1 O)−
2
2 )
22
21
21
−
◦ (O −◦ Q2 O Q22◦ (Q1 O Q1 O)−
2 )
12
22
11
21
◦ Q1 )
Q1 = (Q1 −◦ Q1 ) (Q1 −(Un −◦ Us ) = mm (tr Q1 ) (N12 −◦ S12 )
(Q4 , R4 ) = qrq Us◦ (O I O O)
Q′4 = (I O O O)−−◦ (O O Q4 O)−◦ (O O O I )′Q5 = mm Q12 Q4
◦ Q22
◦ Q21
(Q12
((Q11
6 ), N6 ) = e (N2 , R4 )
6 )
6 −
6 −
′
Q6 = (I O O O)−◦(O Q11
Q12
O)
6
6
21
22
−
◦(O Q6 Q6 O)−◦(O O O I )in (mm Q5 Q′6 , (N1 Un )−◦ (O N6 ))◦ (| − s| |c|)g (a, b)= (|c| |s|)−a−bwhere c = √, s= √
a2 + b 2
a2 + b 2
Note that Nij and Sij (i, j ∈ {1, 2}) have the same shape and Qijk (i, j, k ∈ {1, 2})have the same shape.

3.3. Homomorphism-based Parallel Skeletons for Trees

43

We can efficiently parallelize some parts of these complicated recursive functions,such as matrix multiplication and independent calculations like (Q1 , R1 ) = qrq A11and (Q2 , R2 ) = qrq A21 . These independent calculations are explicitly parallelized
by describing them with the map skeleton as follows.
(Q1 , R1 ) = qrq A11(Q2 , R2 ) = qrq A21⇓|(Q1 , R1 )|−◦ |(Q2 , R2 )| = map apply (|(qrq , A11 )|−◦ |(qrq , A21 )|)Here, the function apply is defined as apply (f, x) = f x. It is, however, still an
open problem whether the complicated recursive functions can be parallelized with
our defined skeletons.

3.3

Homomorphism-based Parallel Skeletons forTrees

We will introduce the following five binary-tree skeletons, which are first proposed by
Skillicorn [Ski96]. These skeletons are basic primitives in the parallel computation
for trees.
• Two node-wise computations: mapb and zipwithb• Two bottom-up computations: reduceb and uAccb (upwards accumulate)• One top-down computation: dAccb (downwards accumulate)In the following, we will give the formal denotational definition of the skeletons as sequential recursive functions, and show the intuitive meaning of the basic
binary-tree skeletons together with additional conditions for parallel implementation [Mat07].

Skeletons for node-wise computations
The skeletons mapb and zipwithb for node-wise computations on trees are defined asfollows.
mapb kl kn (BLeaf n)= BLeaf (kl n)mapb kl kn (BNode l n r) = BNode (mapb kl kn l) (kn n) (mapb kl kn r)
zipwithb kl kn (BLeaf n) (BLeaf n′ ) = BLeaf (kl n n′ )zipwithb kl kn (BNode l n r) (BNode l′ n′ r′ )= BNode (zipwithb kl kn l l′ ) (kn n n′ ) (zipwithb kl kn r r′ )44

3. Homomorphism-based Design of Parallel Skeletons

The parallel skeleton mapb takes two functions kl and kn and a binary tree, andapplies kl to each leaf and kn to each internal node. The parallel skeleton zipwithb
takes two functions kl and kn and two binary trees of the same shape, and zips
the trees up by applying kl to each corresponding pair of leaves and kn to eachcorresponding pair of internal nodes. Since the functions are applied independently
to the nodes, these two skeletons require no additional condition for their parallel
implementation.
The mapb skeleton can be defined in the form of tree homomorphism as follows.
mapb kl kn = ([kl′ , kn′ ])b
where kl′ a= BLeaf (kl a)′
kn l b r = BNode l (kn b) r
The zipwithb skeleton cannot be defined as a tree homomorphism, but we can formalize it with tree anamorphism [MFP91].

Skeletons for bottom-up computations
The skeletons reduceb and uAccb for bottom-up computations on trees are defined
as follows.
reduceb k (BLeaf n)= n
reduceb k (BNode l n r) = k (reduceb k l) n (reduceb k r)
uAccb k (BLeaf n)uAccb k (BNode l n r)

= BLeaf n= let l′ = uAccb k l
r′ = uAccb k r
in BNode l′ (k (root b l′ ) n (root b r′ )) r′

The parallel skeleton reduceb takes a function k and a binary tree, and collapses
the tree into a value by applying the function k to each internal node in a bottomup manner. The parallel skeleton uAccb takes a function k and a binary tree, andapplies the function k in a bottom-up manner while putting the intermediate result
on each node. These two skeletons require an additional condition for the existenceof an efficient parallel implementation of them. The skeletons reduceb and uAccb
called with parameter function k require the existence of four auxiliary functions φ,
ψn , ψl , and ψr satisfying the following three equations.
kxny= ψn x (φ n) y′ψn (ψn n x y) n r = ψn x (ψl n′ n r) y
ψn l n (ψn n′ x y) = ψn x (ψr l n n′ ) y
The reduceb and uAccb skeletons can be defined in the form of tree homomor-

3.3. Homomorphism-based Parallel Skeletons for Trees

45

phism as follows.
reduceb k = ([id , k])b
uAccb k= ([kl′ , kn′ ])b
where kl′ a= BLeaf akn′ l b r = BNode l (k (root b l) b (root b r)) r

Skeleton for top-down computations
The skeleton dAccb for top-down computations on trees is defined as follows.
dAccb (gl , gr ) c (BLeaf n)= BLeaf cdAccb (gl , gr ) c (BNode l n r) = BNode (dAccb (gl , gr ) (gl c b) l) c
(dAccb (gl , gr ) (gr c b) r)
The parallel skeleton dAccb takes a pair of functions gl and gr , a parameterc and a binary tree. This skeleton updates parameter c in a top-down mannerusing gl for the left child and gr for the right child, and puts the parameter c oneach node. The parameter c is called accumulative parameter. This skeleton also
requires an additional condition for an efficient parallel implementation. The dAccb
skeleton called with parameter functions gl and gr requires the existence of auxiliary
functions φl , φr , ψu , and ψd satisfying the following three equations.
gl c n= ψd c (φl n)
gr c n= ψd c (φr n)
ψd (ψd c n) m = ψd c (ψu n m)
The dAccb skeleton can be defined as the following higher-order tree homomorphism.
dAccb (gl , gr ) c = λt.([kl′ , kn′ ])b t cwhere kl′ a= λc′ .BLeaf c′′
kn fl b fr = λc′ .BNode (fl (gl c′ b)) c′ (fr (gr c′ b))Worth noting is the application scope of the binary-tree skeletons under theconditions for parallel implementations. Our conditions cover a wider class of tree
manipulations than those introduced by Skillicorn [Ski94, Ski96] do. For algebraic
tree computations our conditions cover the same class as those studied by Abrahamson et al. [ADKP89], even though they are specified in a different way.
Example programming with the tree skeletons can be found in [Mat07, Ski97],
which includes XPath queries [BBC+ 06] and maximum marking problems [SHTO00,
Bir01] on trees.

46

3. Homomorphism-based Design of Parallel Skeletons

Implementation of Parallel Tree Skeletons
The defined parallel tree skeletons can be implement efficiently in parallel for many
parallel architectures.
The binary tree skeletons have parallel implementation based on tree contraction
algorithms [MR85, ADKP89, RT94, BSWB02, MW97, CV88, DH98, LM88]. Here, let
the model of parallel computers be EREW PRAM with P processors, and N denotethe number of nodes of a tree. We assume that all the functions including auxiliary
functions passed to skeletons are computed sequentially in constant time, and thatthe conditions of the reduceb , uAccb and dAccb are satisfied. The mapb and zipwithb
skeletons can be implemented just by applying functions independently to each node,and they run in O(N/P ) parallel time. The reduceb skeleton can be implemented
by tree contraction algorithms and it runs in O(N/P + log P ) parallel time. The
uAccb and dAccb skeletons are generalized computational patterns of the algebraic
tree computations [ADKP89], and Gibbons et al. [GCS94] developed parallel implementations of them based on tree contraction algorithms. With their implementations, the uAccb and dAccb skeletons can be computed in O(N/P + log P ) paralleltime. In other words, the mapb and zipwithb skeletons achieve linear speedup underP ≤ O(N ), and the reduceb , uAccb , and dAccb skeletons achieve linear speedup
under P ≤ O(N/ log N ).
Although the original tree contraction algorithms have been designed for sharedmemory computers, Matsuzaki [Mat07] has adapted the algorithms for use in distributed parallel machines. With
√ their implementation, the tree skeletons achieve
linear speedup under P ≤ O( N ).
Please refer for the details to Matsuzaki’s thesis [Mat07], because the implementation is too complicated to show here.

Chapter 4
Fusion Optimizations of Parallel
Skeletons
It has been shown so far that compositions of homomorphisms and skeletons provide
us with a powerful mechanism to describe parallel algorithms, where parallelism inthe original parallel algorithms can be well captured. In this chapter, we move on
from issues of parallelism to the issues of efficiency.
Since skeleton programs are developed in the compositional style, they often
have overheads of redundantly many loops and unnecessary intermediate data. To
make skeleton programs efficient, not only each skeleton is implemented efficiently
in parallel, but also optimizations over skeleton compositions are necessary.
The inefficiency problem arises also in sequential programming of the compositional style, and much research has been done deeply to solve the problem in thefield of functional programming. One well known technique to solve the problem
is fusion optimization [Wad88, GLJ93, TM95, Sve02], in which consecutive functions
are fused into one function to remove redundant generation of intermediate data
structures between them.
In this chapter, we will study domain-independent fusion optimizations to improve efficiency of skeleton compositions. Fusion of consecutive skeletons is important and effective, since it can remove redundant generation of intermediate data
structures. It is especially effective in the case that elements of the input arrays havesome structures, such as nested lists and arrays, because the overhead of allocationand deallocation of elements becomes serious and sometimes it becomes the main
reason of inefficiency.
First, we will study fusion optimization of list skeletons, in which the good
fusibility of the homomorphism plays an important role. Then, we will proceed tofusion optimization of skeletons on two-dimensional arrays. After that, using thefusion optimizations, we will show derivation of non-trivial efficient program for the
maximum rectangle sum problem on two-dimensional arrays. A general derivation
strategy with fusions will be given there.

48

4.1

4. Fusion Optimizations of Parallel Skeletons

Useful Fusion Laws for Skeletons on Lists

We will introduce several interesting algebraic laws for fusion of list skeletons. We
will first introduce an important law of homomorphism. Then, we will derive interesting laws for skeletons from the law of homomorphism.

4.1.1

Fusion of List Homomorphism

Homomorphism enjoys many nice transformation rules, among which the following
fusion rule is of particular importance. The fusion rule gives us a way to create a
new homomorphism from composition of a function and a homomorphism, whichplays a key role in derivation of efficient parallel programs.
Lemma 4.1 (Fusion law of list homomorphism [Bir88]). Let g and ([f, ⊕]) be given.
If there exists ⊙ such that for any x and y the equationg (x ⊕ y) = g x ⊙ g y
holds, theng ◦ ([f, ⊕]) = ([g ◦ f, ⊙]) .
Proof. The lemma is proved by induction on the structure of lists.
Here, we will show an example fusion of the program sum ◦ double, which doubles
all elements of a list and sums them up. Its fusion is shown below.
sum ◦ double={ double is a homomorphism }sum ◦ ([[·] ◦ dbl , ++])={ sum (x ++ y) = sum x + sum y, and the fusion lemma }([sum ◦ [·] ◦ dbl , +])={ sum ◦ [·] = id }([dbl , +])This fused function is efficient because it has no intermediate data, while the original
program has intermediate data between two functions sum and double .

4.1.2

Fusion Laws for Skeletons

We will introduce some instances of the homomorphism fusion. Since the fusion
theorem requires us to derive new operator satisfying the condition, automatic useof the theorem is difficult. Thus, we will instantiate the theorem for easy use of
fusions.
We first show an interesting law that shows any homomorphism can be written
as a composition of map and reduce. In other words, a composition of map andreduce can be fused into one homomorphism.

4.1. Useful Fusion Laws for Skeletons on Lists

49

Lemma 4.2 (Homomorphism [Bir88]). A homomorphism ([f, ⊕]) can be written as
a composition of map and reduce:([f, ⊕]) = reduce (⊕) ◦ map f .
Proof. We can prove the lemma with Lemma 4.1, substituting reduce (⊕) andmap f = ([[·] ◦ f, ++]) for the function g and the homomorphism ([f, ⊕]) in the lemma,
respectively.
This lemma implies that if we have efficient parallel implementations for reduceand map, we get an efficient implementation for homomorphism. Conversely, it
implies that we can compute efficiently the composition of two skeletons if we haveefficient implementation for homomorphism.
The following lemma eliminates intermediate data between two maps by fusing
them into a single map.
Lemma 4.3 (Map-map fusion). The following equation holds for any functions f
and g.
map g ◦ map f = map (g ◦ f )Proof. We can prove the lemma with Lemma 4.1, substituting map g and map f =([[·] ◦ f, ++]) for the function g and the homomorphism ([f, ⊕]) in the lemma, respectively.
This lemma is useful because we can fuse a sequence of maps into only one map;
we can write a program step by step with many maps, and we can get efficient
program by fusing them with the lemma.

4.1.3

Accumulate-buildJ Fusion

We will introduce another fusion based on a general form of skeletons on lists [HIT02],
of which foundation is also the homomorphism. The general form is called accumulate
The general form accumulate is equivalent to a specific composition of other skeletons, and thus can represent various computation of skeleton compositions. It isdefined as follows.
accumulate (⊕) p (⊗) q g x c = let es ++ [e] = scan (⊗) ([c] ++ map q x)
ac = zip x es
in reduce (⊕) (map p ac) ⊕ g e
We will denote the accumulate (⊕) p (⊗) q g with special brackets [[g, (p, ⊕), (q, ⊗)]].
The fusion using accumulate is based on the idea of short-cut fusions [Wad88,
GLJ93], in which the generation of intermediate data structures between a producer
and a consumer is canceled by fusing the producer and the consumer. We will50

4. Fusion Optimizations of Parallel Skeletons

use the skeleton accumulate as a consumer, and the following function buildJ as a
producer.
buildJ gen = gen (++) [·] [ ]We can express the list skeletons by using the functions defined above.
map f = buildJ (λc s e. [[λ−.e, (s ◦ f ◦ π1 , c), (−, −)]])
reduce (⊕) = [[λ−.ı⊕ , (π1 , ⊕), (−, −)]]
scan (⊕) x = buildJ (λc s e. [[s, (λ(a, e′ ). s e′ , c), (id, ⊕)]]) x ı⊕
The following theorem gives us the fusion rule to fuse the skeletons describedwith accumulate and buildJ .
Theorem 4.4 (BuildJ-accumulate-buildJ fusion). Provided that function gen havethe type forall α.(α → α → α) → (β → α) → α → α, the following equation holds.
buildJ (λc s e. [[g, (p, ⊕), (q, ⊗)]]) (buildJ gen x) e
= π1 (buildJ (λc s e. gen (⊙) f d) x e)
where (u ⊙ v) e = let (r1 , s1 , t1 ) = u e(r2 , s2 , t2 ) = v (e ⊗ t1 )in (s1 ⊕ r2 , s1 ⊕ s2 , t1 ⊗ t2 )f ae
= (p (a, e) ⊕ g (e ⊗ q a), p (a, e), q a))de= (g e, −, −)A variant of the above fusion rule is the following buildJ-cataJ-buildJ rule.
buildJ (λc s e. [[λ−.φ1 , (φ2 ◦ π1 , φ3 ), (−, −)]]) ◦ (buildJ gen) = buildJ (λc s e.gen φ1 φ2 φ3 )For example, we can fuse two maps into one map using the above rule as follows.
map f ◦ map g= { description of map with buildJ }buildJ (λc s e. [[λ−.e, (s ◦ f ◦ π1 , c), (−, −)]])◦ buildJ (λc s e. [[λ−.e, (s ◦ g ◦ π1 , c), (−, −)]])= { the fusion rule }buildJ (λc s e. [[λ−.e, (s ◦ f ◦ g ◦ π1 , c), (−, −)]])= { description of map with buildJ }map (f ◦ g)

4.2

Useful Fusion Laws for Skeletons on Twodimensional Arrays

We will introduce several interesting algebraic laws for fusion of two-dimensional
array skeletons. We will first introduce an important law of homomorphism. Then,we will derive interesting laws for skeletons from the law of homomorphism.

51

4.2. Useful Fusion Laws for Skeletons on Two-dimensional Arrays

4.2.1

Fusion of Homomorphism for Two-dimensional Arrays

Homomorphism enjoys many nice transformation rules, among which the following
fusion rule is of particular importance. The fusion rule gives us a way to create a
new homomorphism from composition of a function and a homomorphism. As will
be seen in Chapter 4.3, it plays a key role in derivation of efficient parallel programson abide-trees.
Theorem 4.5 (Fusion law of abide-tree homomorphism). Let g and ([f, ⊕, ⊗]) begiven. If there exist ⊙ and ⊖ such that for any x and y the following equations
g (x ⊕ y) = g x ⊙ g y
g (x ⊗ y) = g x ⊖ g yhold, theng ◦ ([f, ⊕, ⊗]) = ([g ◦ f, ⊙, ⊖]) .
Proof. The theorem is proved by induction on the structure of abide-trees.
The base case is shown below.
(g ◦ ([f, ⊕, ⊗])) |a|={ Definition of ([f, ⊕, ⊗]) }g (f a)={ Definition of ([g ◦ f, ⊙, ⊖]) }([g ◦ f, ⊙, ⊖]) |a|
Then, the inductive case for −◦ is shown as follows.
====

is proved similarly.

−◦

The inductive case for(g ◦ ([f, ⊕, ⊗])) (x−◦ y){ Definition of ([f, ⊕, ⊗]) }g (([f, ⊕, ⊗]) x ⊕ ([f, ⊕, ⊗]) y){ Definition of h }g (([f, ⊕, ⊗]) x) ⊙ g (([f, ⊕, ⊗]) y){ Hypothesis of induction }([g ◦ f, ⊙, ⊖]) x ⊙ ([g ◦ f, ⊙, ⊖]) y{ Definition of ([g ◦ f, ⊙, ⊖]) }([g ◦ f, ⊙, ⊖]) (x−
◦ y)

−
◦

Here, we will show an example fusion of the program sum ◦ allsquare, whichsquares all elements of the input and sums them up. Definitions of sum and allsquare
are given as follows: sum = ([id , +, +]) and allsquare = ([| · | ◦ sqr ,−◦, ]), where

52

4. Fusion Optimizations of Parallel Skeletons

sqr x = x2 . Its fusion is shown below.
sum ◦ allsquare{ allsquare is a homomorphism }sum◦, ]) ◦ ([| · | ◦ sqr ,−sum (x−◦ y) = sum x + sum y=
the fusion withsum (x y) = sum x + sum y([sum ◦ | · | ◦ sqr , +, +])= { sum ◦ | · | = id }([sqr , +, +])
−
◦

−
◦

=

This fused function is efficient because it has no intermediate data, while the original
program has intermediate data between two functions sum and allsquare .

4.2.2

Fusion Laws of Skeleton Compositions

Although the fusion law of homomorphism is very strong, there is a problem that wehave to find the new operators ⊙ and ⊖ for the given function g, and this is difficultin general. Therefore, we will derive more easy-to-use fusion laws for skeletons fromthe homomorphism’s fusion law.
The first fusion law shows that any homomorphism can be written as a composition of map and reduce. In other words, the pair of map and reduce can be fused
into a homomorphism.
Lemma 4.6 (Homomorphism). A homomorphism ([f, ⊕, ⊗]) can be written as acomposition of map and reduce:([f, ⊕, ⊗]) = reduce (⊕, ⊗) ◦ map f .
−◦Proof. We can prove the lemma with Theorem 4.5, substituting reduce (⊕, ⊗) andmap f = ([| · | ◦ f,−◦, ]) for the function g and the homomorphism ([f, ⊕, ⊗]) in thestatement, respectively.
The next lemma gives us a way to fuse consecutive maps into one map.
Lemma 4.7 (Map-map fusion). For any functions f and g, two consecutive maps
are fused into one map as follows.
map g ◦ map f = map (g ◦ f )−◦Proof. We can prove the lemma with Theorem 4.5, substituting map g and map f =([| · | ◦ f,−◦, ]) for the function g and the homomorphism ([f, ⊕, ⊗]) in the statement,
respectively.

4.3. Developing Efficient Programs with Parallel Skeletons

53

Using the above two lemmas, we can optimize the following naively composedskeleton program for computing the second moment (variance) with the parameterc againstP
an array. The definition of the second moment of an array a(i,j) is as2
follows:
i,j (a(i,j) − c) . Here, we can use reduce (+, +) to compute the sum,map sqr to take squares, and map (subtract c) to take the subtractions, wheresubtract c x = x − c, and sqr x = x2 .
sndMom c = reduce (+, +) ◦ map sqr ◦ map (subtract c)Now, we perform calculation with the above fusion lemmas.
sndMom c= { Map-map fusion of Lemma 4.7 }reduce (+, +) ◦ map (sqr ◦ subtract c)= { Map-reduce fusion of Lemma 4.6 }([sqr ◦ subtract c, +, +])Now, if we have implementation of the homomorphism, we can compute the second
moment without any intermediate data structure. Or, at least we can compute thesecond moment with only one intermediate data structure using implementations of
reduce and map skeletons. In both cases, the fused program has less intermediate
data structures and thus more efficient than the original naive program.

4.3

Developing Efficient Programs with Parallel
Skeletons

In this section, we will illustrate a strategy to guide programmers to develop efficientparallel algorithms systematically through program transformation with the fusion
laws. Recall homomorphisms have efficient parallel implementation. Thus, a goalof this derivation may be to write a program by a homomorphism. We will focus
on developing parallel programs on two-dimensional arrays, although the strategy
is applicable for other data structures.
Unfortunately, not all functions can be specified by a single homomorphism.
Therefore, we first introduce a more powerful tool called almost-homomorphism[Col95]. After that, we will introduce two important theorems for deriving efficient
programs, one of which is the special case of the fusion law of homomorphism. Then,
we demonstrate the strategy with an example problem.

4.3.1

Almost-homomorphism

Not all functions can be specified by a single homomorphism, but we can always
tuple these functions with some extra functions so that the tupled functions canbe specified by a homomorphism. An almost homomorphism is a composition of a
projection function and a homomorphism.

54

4. Fusion Optimizations of Parallel Skeletons

Definition 4.8 (Almost-homomorphism). A function g is said to be almosthomomorphism, if g is defined as a composition of a homomorphism ([f, ⊕, ⊗]) anda projection function π1 as follows.
g = π1 ◦ ([f, ⊕, ⊗])Since projection functions are simple, almost homomorphisms are suitable forparallel computation as homomorphisms are.
In fact, every function can be represented in terms of an almost homomorphismat the cost of redundant computation. Let k be a nonhomomorphic function, andconsider a new function g such that g x = (k x, x). The tupled function g can be ahomomorphism as follows.

−
◦

−
◦

−
◦

g |a|
= (k |a|, |a|)g (x−◦ y) = g x ⊕ g ywhere (k1 , x1 ) ⊕ (k2 , x2 ) = (k (x1 −◦ x2 ), x1 −◦ x2 )g (x y) = g x ⊗ g ywhere (k1 , x1 ) ⊗ (k2 , x2 ) = (k (x1 x2 ), x1 x2 )Since the first component of the result of g is the result of k, we have the following
equation:k = π1 ◦ g = π1 ◦ ([g ◦ | · |, ⊕, ⊗]) .
This equation shows that k is an almost-homomorphism.
However, the definition above is not efficient because binary operators ⊕ and⊗ do not use the previously computed values k1 and k2 . In order to derive a goodalmost homomorphism, we should carefully define a suitable tupled function, making
full use of previously computed values. We will see this in our parallel program
development later. It is an open problem to determine the class of problems thathave good (efficient) almost-homomorphic implementation.

4.3.2 Two Theorems for Deriving Efficient Parallel Programs
We will introduce two important theorems for deriving efficient parallel programs
using almost-homomorphism.
First, we will propose a way of deriving almost homomorphism from mutual
recursive definitions. For notational convenience, we defineinfi = f1 △ f2 △ · · · △ fn
in 1x(⊕i )y = (x ⊕1 y, x ⊕2 y, . . . , x ⊕n y) .
1

Our main idea is based on the following theorem.

4.3. Developing Efficient Programs with Parallel Skeletons(4.10)−◦

Theorem 4.9 (Tupling).
Let h1 , h2 , . . . , hn be mutual recursively defined by= fi aa, hi |a|ahi (x−◦ y) = ((an1 hi ) x) ⊕i ((an1 hi ) y) ,hi (x y) = (( n1 hi ) x) ⊗i (( n1 hi ) y) .

55

Then

an

1 hi is a homomorphism ([an

1 fi ,an1 ⊕i ,an1 ⊗i ]) .

−◦Proof. The theorem is proved based on the definition of homomorphisms. According
to the definition of array homomorphisms, it is sufficient to prove thataa(an1 hi ) |a|= ( an1 fi ) a , aa(an1 hi ) (x−◦y) = ((an1 hi ) x) (an1 ⊕i ) ((an1 hi ) y) ,( n1 hi ) (x y ) = (( n1 hi ) x) ( n1 ⊗i ) (( n1 hi ) y) .
The first equation is proved by the following calculation.
(an

1 hi ) |a|a{ Definition of }(h1 |a|, . . . , hn |a|)={ Definition of hi }(f1 a, . . . , fn a)a= a { Definition of }( n1 fi ) a=

The second is proved as follows.
(

an

◦ y)
1 hi ) (x−a{ Definition of }(h1 (x−◦ y), . . . , hn (x−◦ y))=of hi }
a{n Definition aaa((( 1 hi ) x) ⊕1 (( n1ahi ) y), . . . , (( n1 hi ) x) ⊕n (( n1 hi ) y))=an{ Definition
an of a}n(( 1 hi ) x) ( 1 ⊕i ) (( 1 hi ) y)=

The third is proved similarly.
Theorem 4.9 says that if function h1 is mutually defined with other functions (i.e.,h2 , . . . , hn ) which traverse over the same array in the specific form of Eq. (4.10), thentupling h1 , . . . , hn will give a homomorphism. It follows that every hi is an almost
homomorphism.
Next, we will give the following theorem showing how to fuse a function withan almost homomorphism to get new another almost homomorphism, which is anextension of the fusion theorem of homomorphism(Theorem 4.5).

56

4. Fusion Optimizations of Parallel Skeletons

Theorem 4.11
Fusion).
a (AlmostaaLet h and ([ n1 fi , n1 ⊕i , n1 ⊗i ]) be given. If there exist ⊙i , ⊖i (i = 1, . . . , n) andH = h1 × h2 × · · · × hn (h1 = h) such that for any i, x and y,hi (x ⊕i y) = H x ⊙i H yhi (x ⊗i y) = H x ⊖i H yhold, thenh◦(π1 ◦([

an

1 fi ,an1 ⊕i ,an1 ⊗i ])) = π1 ◦([an

1 (hi ◦fi ),an

1 ⊙i ,an1 ⊖i ]) .

(4.12)

Proof. The theorem is proved by some calculation and Theorem 4.5.
We prove it by the following calculation.
aaah ◦ (π1 ◦ ([ n1 fi , n1 ⊕i , n1 ⊗i ]))={ Definitionof H andan aa n π1 }nπ1 ◦ H ◦ ([ 1 fi , 1 ⊕i , 1 ⊗i ])={a
Theorem 4.5aand thean proofs below }nnπ1 ◦ ([ 1 (hi ◦ fi ), 1 ⊙i , 1 ⊖i ])To complete the above proof, we need to show

a
an
fi )
= n1 (hi ◦a H ◦ ( a1 fi )nH(x (a1 ⊕i ) y) = (H x) (an1 ⊙i ) (H y)H(x ( n1 ⊗i ) y) = (H x) ( n1 ⊖i ) (H y) .

These equations are proved as follows.
a(H ◦ ( n1 fi )) aa=
{ Definition of and H }((h1 ◦ f1 ) a, . . . , (hn a◦ fn ) a)= a { Definition of }( n1 (hi ◦ fi )) aa
H (x ( n1 ⊕i ) y)a=
{ Definition of and H }(h1 (x ⊕1 y), . . . , hn (x ⊕n y))={ Assumption of hi }((H x) ⊙1 (H y), . . .a
, (H x) ⊙n (H y))={ Definitionof}an(H x) ( 1 ⊙i ) (H y)
The third is similar to the second, and can be proved similarly.
Theorem 4.11 says that we can fuse a function with an almost homomorphism to
get another almost homomorphism by finding h2 , . . . , hn together with ⊙1 , . . . , ⊙n ,⊖1 , . . . , ⊖n that satisfy Eq. (4.12).

4.3. Developing Efficient Programs with Parallel Skeletons

4.3.3

57

A Strategy for Deriving Efficient Parallel Programs

Now, we will give a strategy to derive efficient parallel program using almosthomomorphism and the theorems.
Our strategy for deriving efficient parallel programs on two-dimensional arrays
consists of the following four steps, extending the result of lists [HIT97]. The goalof the strategy is to write a given program by an efficient almost-homomorphismthat has an efficient implementation in parallel.
Step 1. Define the target program p as a composition of p1 , . . . , pn that are alreadydefined, i.e. p = pn ◦ · · · ◦ p1 . Each of p1 , . . . , pn may be defined as acomposition of small functions or a recursive function.
Step 2. Derive an almost homomorphism from the recursive definition of p1 using
the tupling of Theorem 4.9.
Step 3. Fuse p2 into the derived almost homomorphism to obtain a new almost
homomorphism for p2 ◦ p1 , and repeat this derivation until pn is fused. In
this step, we can use the almost-fusion of Theorem 4.11.
Step 4. Let π1 ◦ ([f, ⊕, ⊗]) be the resulting almost homomorphism for pn ◦ · · · ◦ p1 obtained at Step 3. For the functions inside the homomorphism, namely f , ⊕and ⊗, try to repeat Steps 2 and 3 to find efficient parallel implementations
for them.
The key idea is to make a seed almost-homomorphism from p1 by the tupling, andmake the almost-homomorphism bigger by fusing the functions p2 , . . . , pn .

4.3.4

Deriving an Efficient Parallel Program for Maximum
Rectangle Sum Problem

Now, we demonstrate the strategy through a derivation of an efficient program
for the maximum rectangle sum problem: compute the maximum of sums of allrectangle areas in a two-dimensional data. This problem was originated by Bentley [Ben84a,Ben84b] and improved by Takaoka [Tak02]. The solution can be used ina sort of data mining and pattern matching of two dimensional data. For example,
for the following two-dimensional data
3 −1
4 −1 −5 1 −4 −1
5 −3 
−41
5
3
1
the result should be 15, which denotes the maximum sum contributed by the subrectangular area with bold numbers above. To appreciate difficulty of this problem,
we ask the reader to pause for a while to think of how to solve it.

58

4. Fusion Optimizations of Parallel Skeletons

Step 1. Defining a Clear Parallel Program
We can compute the maximum rectangle sum as follows: enumerating all possible
rectangles, then computing sums for all rectangles, and finally returning the maximum value as the result. Therefore, a clear, straightforward program mrs can be
described as follows.

◦
−
◦ −

mrs = max ◦ map sum ◦ rects ′wheremax = reduce (↑, ↑)sum = reduce (+, +)rects ′ = flatten ◦ map TLs ◦ BRs
TLs= scan (−◦, ) ◦ map | · |BRs = scanr (−◦, ) ◦ map | · |Here, rects ′ generates all possible rectangles of the given array, using TLs and BRs togenerate top-left and bottom-right rectangles, respectively. For example, applying
1
2
3, we get the following results.
TLs, BRs, and rects ′ to
4 5 6


 
1
1
2
1
2
3

 
  

1 2 3
1 2
=  1
4 5  4 5 6
 4


3
2 3
1 2 3
1 2 3
6 
5 6
=  4 5 6
BRs
4 5 6
6



 4 5 6  5 6
1
1
2
1
2
3
2
2
3
3
  
   
 

  

 1
3 
2 3
2
1 2 3
1 2
1 2 3
′


= 
rects
6 
5 6
5
4 5 6
4 5
4
4 5 6
4
4 5
4 5 6
5
5 6
6



1 2 3
TLs
4 5 6



Apparently, mrs is a clear parallel program, because it is described with skeletonsand easy to understand. However, it is inefficient in the sense that it needs to executeO(n6 ) addition operations for the input of n × n array. Therefore, we want to develop
a more efficient parallel program according to the strategy.
Step 2. Driving Almost Homomorphism
The second step is to make a seed almost-homomorphism for the following steps.
To this end, we will first do “dilation” of the function rects ′ . Then, we will derive aseed almost-homomorphism.
Although the current mrs is clear to understand, the function rects ′ to generate
all possible rectangles cuts information of rectangles too much to manipulate theprogram further. Therefore, we “dilate” the function rects ′ to hold more information

59

4.3. Developing Efficient Programs with Parallel Skeletonsin the result. The dilation is given as follows (please see Appendix D for the proofof the dilation).

−◦flatten ◦ unwind−◦ ◦ map unwind|a||a| ⊕ reduce ( , ⊕) (map | · | x)
unwind ⊕ (|a| x) ⊕ unwind ⊕ y
−
◦

=
=
=
y)) =

−
◦

◦
−
◦ −

rects ′ = pack ◦ rects
where packunwind ⊕ |a|unwind ⊕ (|a| x)
unwind ⊕ ((|a| x)−◦ (NIL

Here, rects is a dilated function to generate possible rectangles of the given array,and function pack removes information of dilated structures by the function unwind .
An example use of rects is shown below. The return value of rects is an array ofarrays of arrays; the (k, l)-element of the (i, j)-element of the resulting array is a
sub-rectangle consisting of the rows from the ith to the jth and the columns fromthe kth to the lth of the original array. Note that the dilated structure contains
special value NIL to denote blank portions of the above-mentioned array. NIL isseen as a suitable-size array (of arrays) of −∞ (the identity of ↑).


 
 1


 


1 2 3
=
rects

5 6 7






1 2
2




 
  
1 2 3
1 21
  5

5 6 
7 
5 
6




1 2 3


2 3 
2

2 3  


6 
7 
6



3


3


7

 
 

5
5 6
5 6 7
6
6 7  
7

Here is a precise definition of the function rects. Function rects is mutual recursively defined with functions bottoms, tops, rights, and lefts (Figure 4.1) as follows.
rects |a|
= |||a|||rects (x−◦ y) = (rects x

−
◦

−
◦

−
◦

−
◦

rects (x−◦−◦gemm( , zipwith(−◦)) (bottoms x) (tops y))−◦(NIL rects y)y) = zipwith4 fs (rects x) (rects y) (rights x) (lefts y)
where fs s1 s2 r1 l2 = (s1 gemm( , ) r1 l2 )−◦(NIL s2 )where ‘ ’ indicates “don’t care”, and generalized matrix multiplication gemm isdefined as follows1 . Here, we assume that the division of argument arrays are con1

Actually, we have defined the same function gemm(⊕, ⊗) in the previous section in the different
from. But, both definitions returns the same result.

60

4. Fusion Optimizations of Parallel Skeletonssistent.
zipwith(⊕) (g X1 Y1 ) (g X2 Y2 )(g X1 Y )−◦ (g X2 Y )(g X Y1 ) (g X Y2 )|a ⊗ b|
−
◦

−
◦

−
◦

gemm(⊕, ⊗) = g◦ Y2 ) =where g (X1 X2 ) (Y1 −g (X1 −◦ X2 ) Y=
g X (Y1 Y2 )=g |a| |b|=−◦

Figure 4.2 shows recursive computation of rects for the −◦ case. Since indices iand j of the resulting array mean the top row and the bottom row of the generated
rectangles within the original array, the upper-left block (i ≤ mx and j ≤ mx ) of
the resulting array contains rectangles included in the upper block x of the originalarray. Thus, the upper-left block is equal to rects x. Similarly, the lower-right blockis rects y (Figure 4.2-(a)). Each rectangle in the the upper-right block (i ≤ mx andj > mx ) consists of a rectangle touching the bottom of x and a rectangle touching
the top of y as shown in Figure 4.2-(b). Since we have to consider all combinationsof i and j, we compute this block by a general matrix multiplication on bottoms xand tops y as shown in Figure 4.2-(c). The computation of the −◦ case is similar.
Functions bottoms, tops, rights, and lefts are similarly defined as mutual recursive functions shown in Figure 4.1, and their examples are shown in Figure 4.3.
Each of these functions is the partial result of rects in the sense that it returns
the rectangles that are restricted to include some edges of the input as shown in
Figure 4.4. For example, the tops returns the rectangles that include the top edges(the top-most row) of the input array. Similarly, bottoms, rights and lefts returnthe rectangles that include the bottom edges, the right edge and the left edge respectively. The other functions returns the rectangles that include two edges of the
input. Usually, users do not need to understand well these functions because thosewell-used functions are provided by experts.
The function pack gives us the relation between the dilated structure and theoriginal structure. It uses the function unwind to unfold the dilated triangularstructures. Its image is shown below.


a b c
unwind  d e  = (a b c d e f )f−◦

The function unwind similarly unfolds the dilated triangular structures into column
vectors.
It is easily seen the following equation, because the function pack simply changes
the dilated structure and does not change the values, and a reduction of the special
value NIL results in the identity of ↑.
max ◦ map sum ◦ pack = max ◦ map max ◦ map (map sum)The dilation is important for derivation of efficient parallel programs on twodimensional arrays, so that we can successfully derive operators that satisfy the61

4.3. Developing Efficient Programs with Parallel Skeletons

−
◦

−
◦

−
◦

−
◦

−
◦

tops |a|= |||a|||tops (x−◦ y) = tops x map (zipwith(−◦) (cols ′ x)) (tops y)
tops (x y) = zipwith4 ft (tops x) (tops y) (toprights x) (toplefts y)◦ (NIL t2 )where ft t1 t2 tr1 tl2 = (t1 gemm ( , ) tr1 tl2 )−rights |a|= |||a|||rights (x−◦ y) = (rights x

−
◦

−
◦

−
◦

−
◦

bottoms |a|= |||a|||bottoms (x−◦ y) = map (λz. zipwith(−◦) z (cols ′ y)) (bottoms x)−◦ bottoms ybottoms (x y) = zipwith4 fb (bottoms x) (bottoms y) (bottomrights x) (bottomlefts y)
where fb b1 b2 br1 bl2 = (b1 gemm ( , ) br1 bl2 )−◦ (NIL b2 )

−
◦

−
◦

rights (x

−
◦

−
◦

gemm ( , zipwith(−◦)) (bottomrights x) (toprights y))−◦(NIL rights y)y) = zipwith3 fr (rights x) (rights y) (rows ′ y)where fr r1 r2 ro2 = map ( ro2 ) r1 −
◦ r2

−
◦

lefts y)

−
◦

−
◦

−
◦

−
◦

lefts |a|
= |||a|||lefts (x−◦ y) = (lefts x gemm ( , zipwith(−◦)) (bottomlefts x) (toplefts y))−◦ (NIL
lefts (x y) = zipwith3 fl (lefts x) (lefts y) (rows ′ x)
where fl l1 l2 ro1 = l1 map (ro1 ) l2

−
◦

−
◦

−
◦

toprights |a| = |||a|||toprights (x−◦ y) = toprights x map (zipwith(−◦) (right ′ (toprights x))) (toprights y)toprights (x y) = zipwith ftr (toprights x) (toprights y)
where ftr tr1 tr2 = map ( top ′ tr2 ) tr1 −
◦ tr2

−
◦

−
◦

bottomrights |a| = |||a|||bottomrights (x−◦ y) = map (λz. zipwith(−◦) z (top ′ (bottomrights y))) (bottomrights x)−◦bottomrights ybottomrights (x y) = zipwith fbr (bottomrights x) (bottomrights y)
where fbr br1 br2 = map ( top ′ br2 ) br1 −
◦ br2

−
◦

−
◦

−
◦

−
◦

toplefts |a|= |||a|||toplefts (x−◦ y) = toplefts x map (zipwith(−◦) (right ′ (toplefts x))) (toplefts y)
toplefts (x y) = zipwith ftl (toplefts x) (toplefts y)
where ftl tl1 tl2 = tl1 map (right ′ tl1 ) tl2

−
◦

−
◦

−
◦

bottomlefts |a| = |||a|||bottomlefts (x−◦ y) = map (λz. zipwith(−◦) z (top ′ (bottomlefts y))) (bottomlefts x)−◦bottomlefts ybottomlefts (x y) = zipwith fbl (bottomlefts x) (bottomlefts y)
where fbl bl1 bl2 = bl1 map (right ′ bl1 ) bl2

cols ′ y)

−
◦

−
◦

−
◦

−
◦

cols ′ |a|= ||a||
cols ′ (x−◦ y) = zipwith(−◦) (cols ′ x) (cols ′ y)′′cols (x y) = (cols x gemm ( , ) (right (cols ′ x)) (top (cols ′ y)))−◦ (NIL= reduce(≪, ) ◦ map | · |= reduce(≫, ) ◦ map | · |= reduce(−◦, ≫) ◦ map | · |= reduce(−◦, ≪) ◦ map | · |
−
◦ −
◦

top
bottomrightleftrows ′ y)

−
◦

−
◦ −
◦

−
◦

rows ′ |a|= ||a||◦) (right (rows ′ x)) (top (rows ′ y)))−◦ (NILrows ′ (x−◦ y) = (rows ′ x gemm ( ,−′′rows (x y) = zipwith( ) (rows x) (rows ′ y)top ′bottom ′right ′
left ′= the ◦top
= the ◦bottom= the ◦right= the ◦left

Figure 4.1. Auxiliary functions for rects.

62

4. Fusion Optimizations of Parallel Skeletons

!

n
mx

x

mx

y

j
mx
mx re ts x

=

re ts

Z

!
i

my re ts y
my
(a)

x

i

rx
ry

j

y
k

l
(b)

Z

=



tops ybottoms x( )Figure 4.2. Recursive computation of rects.

63

4.3. Developing Efficient Programs with Parallel Skeletons

0
 
B

 B a
B
a b
B
re ts
d B
B


a b
b

=



11

0  
B
B


a b
C
d C
CC
 
b AC
C
d  C
C
A
d
d

a

 

00  0  
11
0
a
a b


 

B
CC
a b B
B
B a a b d CC
tops


b
b AA
d

=


bottoms



=

lefts

d


toplefts

 

a b
=
d

a



a b

0  


ols

a b
d



B
B


=

a



a b

11

B
C
d C
CC
B  
A

A
b

=

d

  

1

a b
d C
C
 
b A
d

a b
d

0



= a





11

a b
CC
 d CCb AC
Cd  CCAdd  

a

a b

a

a b
d

a b
BB
C
d C
CC

 BB  
B
C
A

b
a b B
C
bottomrights=Bd B  d  C
C

A
d
d




bottomlefts

rows

a b
d



a b
d
0



0  

=


 a b

=

a




1

Figure 4.3. Examples of auxiliary functions for rects.

1

a b
d A
d

a b
d A
d

1

a b
d A

d

1 1

00

0

0




a b B
B a btoprights
b
d 

BB
 BB


a b B
=B
d B
B  


d

0
11
0
a b


B
C
B
d C
B  
CC

 B a b

AC
b
b
a b B
C
B
rightsCd B
B d  C
A

d

a

64

4. Fusion Optimizations of Parallel Skeletons

t

tr

l

s

r

bl

b

br

tl
r

Figure 4.4. Corresponding parts of elements in the tuple.

abide property. Usually, operands of operators with the abide property should havecapability for computation in two directions. The dilated information is very usefulto create the capability.
Now, our program is described as follows, and our goal of this step is to make thefirst function rects an almost-homomorphism. The tupling theorem (Theorem 4.9)
gives us a systematic way to execute this step of the strategy.

mrs = max ◦ map max ◦ map (map sum) ◦ rects

We apply the tupling theorem to derive an almost homomorphism for rects. The
definition of rects and the extra functions are in the form of Eq. (4.10). Thus, we canobtain the almost homomorphism shown in Figure 4.5 by tupling these functions.
The operators are straightforward rewriting of the definition of rects.

Step 3. Fusing with Almost Homomorphisms
We aim to derive an efficient almost homomorphism for mrs. The almost-fusion
theorem (Theorem 4.11) gives us a systematic way to execute this step the strategy.
Now, we will apply this theorem to mrs repeatedly.
The second function p2 of our example is map (map sum), so h1 = map (map sum).

65

4.3. Developing Efficient Programs with Parallel Skeletons

rects = π1 ◦ (i111hi ) = π1 ◦ ([

i11
1

i11

fi ,

1

⊕i ,

i11
1

⊗i ])

−
◦

−
◦ −
◦

−
◦

−
◦

−
◦

◦ −
−
◦ −
◦

−
◦ −
◦

where
a11
1 fi |a| = (|||a|||, |||a|||, |||a|||, |||a|||, |||a|||, |||a|||, |||a|||, |||a|||, |||a|||, ||a||, ||a||)a(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 111 ⊕i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where◦)) b1 t2 )−◦ (NIL s2 )s0 = (s1 gemm ( , zipwith(−t0= t1 map (zipwith(−◦) c1 ) t2
◦ b2
b0 = map (λz. zipwith(−◦) z c2 ) b1 −r0 = (r1 gemm ( , zipwith(−◦)) br1 tr2 )−◦ (NIL r2 )l0
= (l1 gemm ( , zipwith(−◦)) bl1 tl2 )−◦ (NIL l2 )◦) (right ′ tr1 )) tr2
tr0 = tr1 map (zipwith(−◦ br2
br0 = map (λz. zipwith(−◦) z (top ′ br2 )) br1 −tl0 = tl1 map (zipwith(−◦) (right ′ tl1 )) tl2◦ bl2
bl0 = map (λz. zipwith(−◦) z (top ′ bl2 )) bl1 −c0 = zipwith(−◦) c1 c2◦) (right ro1 ) (top ro2 ))−◦ (NIL ro2 )ro0 = (ro1 gemm ( ,−

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

a
(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 111 ⊗i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where
s0 = zipwith4 fs s1 s2 r1 l2
where fs s1 s2 r1 l2 = (s1 gemm ( , ) r1 l2 )−◦ (NIL s2 )t0
= zipwith4 ft t1 t2 tr1 tl2
where ft t1 t2 tr1 tl2 = (t1 gemm ( , ) tr1 tl2 )−◦ (NIL t2 )b0 = zipwith4 fb b1 b2 br1 bl2◦ (NIL b2 )where fb b1 b2 br1 bl2 = (b1 gemm ( , ) br1 bl2 )−r0 = zipwith3 fr r1 r2 ro2
where fr r1 r2 ro2 = map ( ro2 ) r1 −◦ r2
l0
= zipwith3 fl l1 l2 ro1
where fl l1 l2 ro1 = l1 map (ro1 ) l2
tr0 = zipwith ftr tr1 tr2◦ tr2
where ftr tr1 tr2 = map ( top ′ tr2 ) tr1 −br0 = zipwith fbr br1 br2
where fbr br1 br2 = map ( top ′ br2 ) br1 −◦ br2
tl0 = zipwith ftl tl1 tl2
where ftl tl1 tl2 = tl1 map (right ′ tl1 ) tl2
bl0 = zipwith fbl bl1 bl2
where fbl bl1 bl2 = bl1 map (right ′ bl1 ) bl2
c0 = (c1 gemm ( , ) (right c1 ) (top c2 ))−◦ (NIL c2 )ro0 = zipwith( ) ro1 ro2

Figure 4.5. Almost-homomorphism definition of rects.

66

4. Fusion Optimizations of Parallel Skeletons

Then, we calculate h1 (x ⊕1 y) to find other functions and operators.
h1 (x ⊕1 y){ Expand x, y and h1 }map (map sum) ((s1 gemm( , zipwith(−◦)) b1 t2 )−◦ (NIL s2 ))={ Definition of map }(map (map sum)s1 map (map sum) (gemm( , zipwith(−◦)) b1 t2 ))−◦(NIL map (map sum)s2 )={ Promotion of map, folding }(h1 s1 gemm( , zipwith(+))(map (map sum) b1 ) (map (map sum) t2 ))−◦(NIL h1 s2 )
−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

In the last formula, functions applied to t1 and b1 should be h2 and h3 , respectively,
which suggests us to define h2 , h3 and ⊙1 as follows.

s2 )

−
◦

−
◦

h1 = h2 = h3 = map (map sum)(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 )⊙1 (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )◦ (NIL= (s1 gemm ( , zipwith(+)) b1 t2 )−Similarly, we can derive ⊖1 by calculating h1 (x ⊗1 y) as follows.

s2 )

−
◦

−
◦

(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 )⊖1 (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= zipwith4 fs s1 s2 r1 l2
where fs s1 s2 r1 l2 = (s1 gemm( , +) r1 l2 )−◦(NILDeriving the other functions and operators by doing similar calculation about ⊕iand ⊗i , we finally get the program shown in Figure 4.6. In this derivation, thefunction H appeared in Theorem 4.11 is as follows:
H = h × h × h × h × h × h × h × h × h × (map sum) × (map sum)
where h = map (map sum) .
Some calculation rules used in this derivation are listed in Appendix A.
We will repeat application of the theorem to our example. Applying the theorem again with the third function p3 = map max , we obtain another almosthomomorphism shown in Figure 4.7 with H = (map max ) × id × id × id × id ×
id × id × id × id × id × id .
Finally, applying such fusion with max will yield the result shown in Figure 4.8.
The function H for the final fusion is as follows:

−
◦

−
◦

H =max × (reduce ( , zipwith(↑))) × (reduce (zipwith(↑), )) × (map (reduce (↑, )))× (map (reduce ( , ↑))) × (reduce ( , )) × (reduce ( , ))× (reduce ( ,−◦)) × (reduce (−◦, )) × id × id

67

4.3. Developing Efficient Programs with Parallel Skeletonsmap (map sum) ◦ rects = π1 ◦ ([

i11
1

fi′ ,

i11
1

⊙i ,

i11
1

⊖i ])

−
◦

−
◦ −
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

◦ −
−
◦ −
◦

−
◦ −
◦

where
a11 ′
1 fi |a| = (||a||, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, |a|, |a|)a(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 111 ⊙i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where◦ (NIL s2 )s0 = (s1 gemm ( , zipwith(+)) b1 t2 )−t0= t1 map (zipwith(+) c1 ) t2
◦ b2
b0 = map (λz. zipwith(−◦) z c2 ) b1 −r0 = (r1 gemm ( , zipwith(+)) br1 tr2 )−◦ (NIL r2 )l0
= (l1 gemm ( , zipwith(+)) bl1 tl2 )−◦ (NIL l2 )tr0 = tr1 map (zipwith(+) (right ′ tr1 )) tr2◦ br2
br0 = map (λz. zipwith(+) z (top ′ br2 )) br1 −tl0 = tl1 map (zipwith(+) (right ′ tl1 )) tl2◦ bl2
bl0 = map (λz. zipwith(+) z (top ′ bl2 )) bl1 −c0 = zipwith(+) c1 c2◦ (NIL ro2 )ro0 = (ro1 gemm ( , +) (right ro1 ) (top ro2 ))−a11(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 1 ⊖i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where
s0 = zipwith4 fs s1 s2 r1 l2
where fs s1 s2 r1 l2 = (s1 gemm ( , +) r1 l2 )−◦ (NIL s2 )t0
= zipwith4 ft t1 t2 tr1 tl2
where ft t1 t2 tr1 tl2 = (t1 gemm ( , +) tr1 tl2 )−◦ (NIL t2 )b0 = zipwith4 fb b1 b2 br1 bl2◦ (NIL b2 )where fb b1 b2 br1 bl2 = (b1 gemm ( , +) br1 bl2 )−r0 = zipwith3 fr r1 r2 ro2◦ r2
where fr r1 r2 ro2 = map (+ro2 ) r1 −l0
= zipwith3 fl l1 l2 ro1
where fl l1 l2 ro1 = l1 map (ro1 +) l2
tr0 = zipwith ftr tr1 tr2◦ tr2
where ftr tr1 tr2 = map (+top ′ tr2 ) tr1 −br0 = zipwith fbr br1 br2◦ br2
where fbr br1 br2 = map (+top ′ br2 ) br1 −tl0 = zipwith ftl tl1 tl2
where ftl tl1 tl2 = tl1 map (right ′ tl1 +) tl2
bl0 = zipwith fbl bl1 bl2
where fbl bl1 bl2 = bl1 map (right ′ bl1 +) bl2
c0 = (c1 gemm ( , +) (right c1 ) (top c2 ))−◦ (NIL c2 )ro0 = zipwith(+) ro1 ro2

Figure 4.6. Derived efficient program of map (map sum) ◦ rects.

68

4. Fusion Optimizations of Parallel Skeletons

map max ◦ map (map sum) ◦ rects = π1 ◦ ([i11
1fi′′ ,

i11
1

⊙′i ,

i11
1

⊖′i ])

−
◦

−
◦ −
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

◦ −
−
◦ −
◦

−
◦ −
◦

where
a11 ′′
1 fi |a| = (|a|, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, ||a||, |a|, |a|)a′(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 111 ⊙i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where◦ (NIL s2 )s0 = (s1 map max (gemm ( , zipwith(+)) b1 t2 ))−t0= t1 map (zipwith(+) c1 ) t2
◦ b2
b0 = map (λz. zipwith(−◦) z c2 ) b1 −r0 = (r1 gemm ( , zipwith(+)) br1 tr2 )−◦ (NIL r2 )l0
= (l1 gemm ( , zipwith(+)) bl1 tl2 )−◦ (NIL l2 )tr0 = tr1 map (zipwith(+) (right ′ tr1 )) tr2◦ br2
br0 = map (λz. zipwith(+) z (top ′ br2 )) br1 −tl0 = tl1 map (zipwith(+) (right ′ tl1 )) tl2◦ bl2
bl0 = map (λz. zipwith(+) z (top ′ bl2 )) bl1 −c0 = zipwith(+) c1 c2◦ (NIL ro2 )ro0 = (ro1 gemm ( , +) (right ro1 ) (top ro2 ))−a11 ′(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 1 ⊖i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where
s0 = zipwith4 fs s1 s2 r1 l2
where fs s1 s2 r1 l2 = s1 ↑ max (gemm ( , +) r1 l2 ) ↑ s2
t0
= zipwith4 ft t1 t2 tr1 tl2
where ft t1 t2 tr1 tl2 = (t1 gemm ( , +) tr1 tl2 )−◦ (NIL t2 )b0 = zipwith4 fb b1 b2 br1 bl2◦ (NIL b2 )where fb b1 b2 br1 bl2 = (b1 gemm ( , +) br1 bl2 )−r0 = zipwith3 fr r1 r2 ro2◦ r2
where fr r1 r2 ro2 = map (+ro2 ) r1 −l0
= zipwith3 fl l1 l2 ro1
where fl l1 l2 ro1 = l1 map (ro1 +) l2
tr0 = zipwith ftr tr1 tr2◦ tr2
where ftr tr1 tr2 = map (+top ′ tr2 ) tr1 −br0 = zipwith fbr br1 br2◦ br2
where fbr br1 br2 = map (+top ′ br2 ) br1 −tl0 = zipwith ftl tl1 tl2
where ftl tl1 tl2 = tl1 map (right ′ tl1 +) tl2
bl0 = zipwith fbl bl1 bl2
where fbl bl1 bl2 = bl1 map (right ′ bl1 +) bl2
c0 = (c1 gemm ( , +) (right c1 ) (top c2 ))−◦ (NIL c2 )ro0 = zipwith(+) ro1 ro2

Figure 4.7. Derived efficient program of map max ◦ map (map sum) ◦ rects

4.3. Developing Efficient Programs with Parallel Skeletons

mrs = π1 ◦ ([i11
1fi′′′ ,i11
1⊙′′i ,i11
1

69⊖′′i ])

−
◦ −
◦

−
◦

◦
−
◦ −

−
◦

◦
◦ −
−
◦ −

◦
−
◦ −

−
◦

−
◦

◦ −
−
◦ −
◦

where
a ′′′
( 11
1 fi ) |a| = (a, |a|, |a|, |a|, |a|, |a|, |a|, |a|, |a|, |a|, |a|)a′′(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 111 ⊙i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where
s0 = (s1 ↑ max (zipwith(+) b1 t2 ) ↑ s2
t0 = zipwith3 ft t1 c1 t2
where ft t1 c1 t2 = t1 ↑ (c1 + t2 )b0 = zipwith3 fb b1 c2 b2
where fb b1 c2 b2 = (b1 + c2 ) ↑ b2
r0 = (r1 gemm (↑, +) (tr br1 ) tr2 )−◦ (NIL r2 )l0 = (l1 gemm (↑, +) bl1 (tr tl2 ))−◦ (NIL l2 )tr0 = tr1 mapc (zipwith(+) (right tr1 )) tr2
br0 = mapc (zipwith(+) (left br2 )) br1 br2◦ mapr (zipwith(+) (bottom tl1 )) tl2
tl0 = tl1 −◦ bl2
bl0 = mapr (zipwith(+) (top bl2 )) bl1 −c0 = zipwith(+) c1 c2
ro0 = (ro1 gemm( , +) (right ro1 ) (top ro2 ))−◦ (NIL ro2 )a11 ′′(s1 , t1 , b1 , r1 , l1 , tr1 , br1 , tl1 , bl1 , c1 , ro1 ) ( 1 ⊖i ) (s2 , t2 , b2 , r2 , l2 , tr2 , br2 , tl2 , bl2 , c2 , ro2 )= (s0 , t0 , b0 , r0 , l0 , tr0 , br0 , tl0 , bl0 , c0 , ro0 )where
s0 = s1 ↑ max (zipwith(+) r1 l2 ) ↑ s2
t0 = (t1 gemm (↑, +) tr1 tl2 )−◦ (NIL t2 )b0 = (b1 gemm (↑, +) br1 bl2 )−◦ (NIL b2 )r0 = zipwith3 fr r1 r2 ro2
where fr r1 r2 ro2 = (r1 + ro2 ) ↑ r2
l0 = zipwith3 fl l1 l2 ro1
where fl l1 l2 ro1 = l1 ↑ (ro1 + l2 )◦ tr2
tr0 = mapr (zipwith(+)(top tr2 )) tr1 −◦ br2
br0 = mapr (zipwith(+)(top br2 )) br1 −tl0 = tl1 mapc (zipwith(+) (right tl1 )) tl2
bl0 = bl1 mapc (zipwith(+) (right bl1 )) bl2
c0 = (c1 gemm( , +) (right c1 ) (top c2 ))−◦ (NIL c2 )ro0 = zipwith(+) ro1 ro2

Figure 4.8. The final efficient program of maximum rectangle sum.

70

4. Fusion Optimizations of Parallel Skeletons

x1

x1

s1

r1

br 1

b1
i

x2

+

t2

j

s2

(a)

max

k

+

tr 2
r2

x2

j

i

(b)

Figure 4.9. Computation of the operator for derived mrs.

In the final program, each of the eleven elements of the resulting tuple is thepartial answer of its counterpart in Figure 4.4. Some parts are calculated with ageneral matrix multiplication, and others are updated with map and zipwith. For
example, the element s0 , which is the solution of the maximum rectangle sum, iseither the maximum of the solutions of upper and lower subarray, or the maximumof the solutions generated by combining partial answers of the top and the bottom
rectangles, as shown in Figure 4.9-(a). Here, the rectangles that share the same edge
are combined by zipwith(+). Similarly, some elements of the array r0 , which is thepartial solutions of the rectangles on the right edge, are calculated with a generalmatrix multiplication as shown in Figure 4.9-(b). Here, (i, j) element in the blockis the maximum of br 1 (i, k) + tr 2 (k, j) for all k.
Provided that we divide the input array into two parts evenly, this final parallel
program uses only O(n3 ) addition operations as follows. For an n × n input array,
the program’s cost T (n, n) satisfies the next equation with gemm’s cost Tgemm (n, n)
and some constants c1 and c2 .
T (n, n) = 4T (n/2, n/2) + c1 Tgemm (n/2, n/2) + c2 n2Since the cost of the general matrix multiplication (gemm) is O(n3 ), the answer of
the above equation is T (n, n) = O(n3 ). This is much better than the initial one.
Moreover, since the general matrix multiplication gemm(↑, +) used in the programis a distance matrix multiplication, we can achieve subcubic cost with the special
implementation of the gemm(↑, +) used in Takaoka’s algorithm [Tak02]. However,
the implementation is somewhat tricky, so that we cannot describe it with ourskeletons.
Step 4. Optimizing Inner Functions
For our example, we may proceed to optimize the operators and functions such asfi′′′ , ⊙′′i and ⊖′′i in the program of Step 3. Since they cannot be made efficient any
more, we finish our derivation of an efficient parallel program.

4.4. Related Work

4.4

71

Related Work

There have been several studies on the optimizations over multiple skeletons based on
fusion transformations [GWL99, WL98, HIT02, MKI+ 04, GS06], which were studied
in depth in the field of functional programming [Wad88, GLJ93]. In particular,
general fusion optimizations on lists [GWL99,WL98,HIT02,MKI+ 04] have achieved
good results both in theory and in practice.
Wedler and Lengauer [WL98] proposed fusion rules on a computation patterncalled parallel linear recursion. Their parallel linear recursion can perform almost
equivalent computation to accumulate, except that parallel linear recursion explicitly
return a pair of a value and a list to allow arbitrary number of compositions. Their
formalization is close to cellular automaton.
Gorlatch et al. [GWL99] proposed a set of fusion rules for skeletons. Their rules
can deal with similar skeleton compositions that the above research can handle,
although their rules are, if anything, case by case.

Chapter 5
Domain-Specific Optimization for
Skeleton Programs
Although the general fusion optimizations so far are reasonably powerful, there
is still large room for further optimizations. Due to the generality of the fusiontransformations, some overheads in skeleton programs are left through the fusion
optimizations. Also, effective optimization of complicated skeleton compositions byfusion often needs human insights to make efficient operators or functions used in
the fused results.
The above problem can be solved when we make fusion optimizations specific tosome domain, exploiting knowledge of the domain. This specific fusion optimization
may include fusion rules specific to the domain, and shortcuts to the fused resultswith human insights.
In this chapter, we will concentrate on optimization of skeleton programs forcomputation involving neighbor elements. The computation can be categorized into
two types: the computation involving a finite number of neighbor elements, suchas filtering of sequences and images, the finite difference method, and some matrixvector operations; and the computation involving an infinite number of neighborelements, such as queries of interesting segments on lists and rectangles (sub-arrays)
on two-dimensional arrays.
First, we will develop domain-specific fusion rules for computation involving afinite number of neighbor elements. To this end, we will introduce a new strategy
for developing domain-specific fusion optimization of skeleton programs. Then, we
will develop the fusion optimization demonstrating the strategy.
Next, we will proceed to optimization of skeleton programs involving an infinite
number of elements. To this end, we will formalize the computation by nested
reductions, and develop shortcut theorems to provide efficient algorithms to nested
reductions by fusion.

74

5. Domain-Specific Optimization for Skeleton Programs

5.1

General Strategy for Domain-Specific Fusion
Optimization

In skeletal parallel programming, domain-specific programs are often developed witha fixed set of skeletons composed in a specific manner. Based on this observation,
we propose the following strategy for developing domain-specific optimizations.
1. Designing a normal form that abstracts target specific computations.
2. Developing specific fusion rules that can completely transform skeleton compositions of specific patterns into the normal forms.
3. Providing efficient implementations of the normal form.
In designing a normal form, we should have the following requirements in mind.
A normal form is specified to describe any computation of target programs but
should not be too general. A normal form should be specific to the target programs,
and should enable us to develop efficient implementation for it. In addition, a normal
form should be closed under the fusion rules to maintain the result of optimizationin the form.
Once we formalize a normal form with fusion rules and efficient implementation,
we can perform the optimization easily: we first transform a skeleton program into
the normal form with the fusion rules, and then we dispatch a suitable efficient
implementation to the obtained normal form, in which the implementation may be
selected according to some mathematical properties of parameters in the normalform.
The big difference from the existing general fusions is that our domain-specificfusions are closed under the target compositions. A big problem of the general
fusions is that no one know how widely the general fusions can fuse skeletons in thegiven programs. They sometimes can not fuse any consecutive skeletons successfully.
For example, no proposed general fusion can fuse parallel skeletons zip and reduce.
Another difference is that the optimized program results in implementation specific to the domain of target programs. The general fusions use general computation
patterns as the results of fusions, which may cause significant inefficiency due totheir generality.

5.2

Computation Involving a Finite Number of
Neighbor Elements

We demonstrate our strategy by developing optimization of skeleton programs forcomputation that involves neighbor elements, which is often seen in scientific computations. In this section, we will concentrate on the computation for one-dimensional
data structures, and use skeleton on lists shown in Chapter 2.

5.2. Computation Involving a Finite Number of Neighbor Elements

75

First, we will define the target domain as a pattern of skeleton compositions.
Next, we will design the normal form to describe computations of the domain. Then,we will develop domain-specific fusion that transforms skeleton compositions of thespecific pattern into normal forms. After that, we will design parallel implementation
for the normal form. Finally, we will extend the target domain to handle reductions
and accumulations.
The optimization developed in this section mainly focuses on domain-specificfusions, and has only one pattern of efficient implementations for the normal form.
We will develop another optimization in the next section, which will focus on various
theorems used to dispatch efficient implementations for the normal form under someconditions on parameters.

5.2.1

Target Skeleton Composition Patterns

Our targets are skeleton programs that involve neighbor elements using combinationof shift≪, shift≫, zip and map. We also deal with skeleton programs that perform one
accumulation or one reduction by scan′ (scanr′ ) or reduce on the result of the above
programs. We define the former program as Program, a program with accumulationas ProgramS , and a program with reduction as ProgramR . A program of ProgramS
can have arbitrary number of map after scan′ or scanr′ .
data Program α = map (β → α) (Program β)| shift≪ α (Program α)| shift≫ α (Program α)| zip (Program β) (Program γ)| [α]data ProgramS α = scan (α → α → α) α (Program α)| scanr (α → α → α) α (Program α)| map (β → α) (ProgramS β)
data ProgramR α = reduce (α → α → α) (Program α)Here, Program α and ProgramS α are programs that generate lists of elements oftype α, while ProgramR α is a program that generates a value of type α. Type β in
map is bound locally. Types β and γ in zip are bound by the relation α = (β, γ).
For simplicity, we do not use binding of variables in the above skeleton programs.
Since the above skeleton programs are inputs for optimization algorithms, we distinguish skeletons in the above skeleton programs from skeletons used in algorithms
by attaching underlines.
Evaluation of the above defined program is given by the following evalP , evalPS

76

5. Domain-Specific Optimization for Skeleton Programsand evalPR , which evaluate the programs straightforwardly.
evalP :: Program α → [α]evalP (map f x) = map f (evalP x)
evalP (shift≪ e x) = shift≪ e (evalP x)
evalP (shift≫ e x) = shift≫ e (evalP x)
evalP (zip y z) = zip (evalP y) (evalP z)
evalP (x)=xevalPS :: ProgramS α → [α]evalPS (scan (⊕) e x) = scan′ (⊕) e (evalP x)
evalPS (scanr (⊕) e x) = scanr′ (⊕) e (evalP x)
evalPS (map f x)
= map f (evalPS x)
evalPR :: ProgramR α → αevalPR (reduce (⊕) x) = reduce (⊕) (evalP x)
In the rest of this section, we mainly focus on Program α to explain our idea ofthe domain-specific fusion. Then, we discuss ProgramS α and ProgramR α in Section
5.2.5 as extensions of the optimization of Program α.
A Running Example
An example target problem is computation of a numerical solution of differential
equations by difference methods, in which each elements of the solution is computed
form its neighboring elements. We will use a simple program for difference method
as our running example.
Let’s consider the following wave-equation as a concrete differential equation.
∂u
∂u
= −C
∂t
∂x
This equation describes propagation of waves. To calculate the propagation of waves,
we replace differential terms by differences. The value of u at time n and at location
i is denoted by uni .

un+1
− uni
−C
i
a−2 uni−2 + a−1 uni−1 + a0 uni + a1 uni+1=∆t∆xwhere (a−2 , a−1 , a0 , a1 ) = (1/6, −1, 1/2, 1/3)
Here, we use a difference computed from two elements on the left and an element on
the right. Rearranging the above equation, we get the following recurrence equation.
un+1= c−2 uni−2 + c−1 uni−1 + c0 uni + c1 uni+1i
Now, let’s consider a program next to compute the values at the next time fromthe current values of u. Here, we will use simple boundary conditions: un−1 = bl0 ,5.2. Computation Involving a Finite Number of Neighbor Elements

77

un0 = bl1 and unN +1 = br for a fixed N . We can implement next using skeletons asfollows.
′
next u = let v−2= map (c−2 ×) (shift≫ bl0 (shift≫ bl1 u))′v−1= map (c−1 ×) (shift≫ bl1 u)′
v0 = map (c0 ×) u
v1′ = map (c1 ×) (shift≪ br u)′′v− = map add (zip v−2
v−1
)
′
′
v+ = map add (zip v0 v1 )in map add (zip v− v+ )′′Here, intermediate variables v−2, v−1, v0′ , and v1′ are used for readability; we caneasily remove these variables to make the program fit to the composition pattern
Program of target programs. The correspondence of the program next and the above′recurrence equation is as follows. First, we generate a list v−2
corresponding to ansequence of the first terms c−2 ui−2 . To this end, we first apply two shift≫ to shift
the elements twice to the right, so that the shifted lists has the value uni−2 at theposition of uni in the input list u. Then, we use map to multiply the coefficient c−2 tothe elements. Figure 5.1 illustrates the arrangement of shifted elements. Similarly,
lists corresponding to the second through the fourth terms are generated by using
shift≪, shift≫ and map. Then, zipping these four lists by zip and adding elements
by map add , we obtain the final result, of which ith element is c−2 ui−2 + c−1 ui−1 +c0 ui + c1 ui+1 .
In the following sections, we will explain our idea by using this example program
next.

5.2.2

Normal Form for the Domain

The first step of our strategy is to design a normal form that can describe any
computation of target programs. In this section, we formalize a normal form that
describes the computation of Program α. The objective of this normal form is to holdnecessary information to compute the result, which will be collected from skeleton
compositions by domain-specific fusions.
Example of Normal Form
Let’s consider a single-loop computation of the running example next. By a simple
observation, its computation can be divided into three parts according to forms ofcomputations of elements: the computation of the center elements, which involves
only elements of the given list; the computation of elements on the left edge, which
involves constants introduced by shift≫; and the computation of elements on theright edge, which involves constants introduced by shift≪. Figure 5.1 illustrates thethree parts.

78

5. Domain-Specific Optimization for Skeleton Programs

In the computation of next, each element in the center part is computed by thefollowing expression. Here, u denotes the element concerned, u≪2 denotes its second
left element, u≪1 its left element, and u≫1 its right element.
ce = “add (add (c−2 ×u≪2 , c−1 ×u≪1 ), add (c0 ×u, c1 ×u≫1 ))”This expression is common among the computation of elements in the center part.
On the other hand, the leftmost element is computed by the following expression.
−→
Here, u[k] denotes the kth element of u from the left edge.
−→−→
l1 = “add (add (c−2 ×bl0 , c−1 ×bl1 ), add (c0 ×u[0], c1 ×u[1]))”−→−→This computation involves variables (u[0] and u[1]) and constants (bl0 and bl1 ) introduced by shift≫. Similarly, computation of the second element and the rightmost←−
element involves constants. Here, u[k] denotes kth element of u from the right edge.
−
→
−
→
−
→
l2 = “add (add (c−2 ×bl1 , c−1 ×u[0]), add (c0 ×u[1], c1 ×u[2]))”
←
−
←
−
←
−
r1 = “add (add (c−2 ×u[2], c−1 ×u[1]), add (c0 ×u[0], c1 ×br ))”Summarizing these observations, whole computation of next can be denoted by
the following triple of expressions, i.e., computation trees.
• The list of computation trees for elements on the left edge, i.e., [l1 , l2 ].
• The common computation tree for elements in the center part, i.e., ce.
• The list of computation trees for elements on the right edge, i.e., [r1 ].
Figure 5.2 shows these computation trees. To obtain the result from this triple, we
compute each element on the edges by its computation tree, and compute elementson the center part by the common computation tree against indices in a single loop.
Generally, such a triple denotes a computation of a target skeletal program.
Thus, in the next section, we formalize this triple as a normal form of our target
skeletal programs. Transformation of a skeleton program into a normal form is latershown in Section 5.2.3.
Definition of Normal Form for the Domain
We will define structure of our normal form. A normal form is defined as a triple ofthe following three as argued so far: a list of computation trees for elements on theleft edge, a common computation tree for the center part, and a list of computation
trees for elements on the right edge.
type NForm α = ([Tree α], Tree α, [Tree α])
We denote this triple with special brackets like [[ ls, zms, rs ]]. Here, ls is the list ofcomputation trees for the left edge, zms is the common computation tree for the79

5.2. Computation Involving a Finite Number of Neighbor Elements

shift≫ b0 (shift≫ b1 u)

b0shift≫ b1 u

b1

−
→
u[0]
−
→
u[1]
−
→
u[2]

−
→
u[0]
−
→
u[1]

u
shift≪ b2 u

[[

[

l1

−−−→
u[i − 2]−−−→u[i − 1]
−
→
u [i]
−−−→
u[i + 1]

b1

l2 ]

,

←
−
u[2]
←
−
u[1]
←
−
u[0]
b2

ce

computation withintroduced constants[common computation
except for the index ir1 ] ]]
computation withintroduced constants

Figure 5.1. Lists arranged by shifts and regions of elements calculated by triples fornext. A list of computation trees for elements on the left edge [l1 , l2 ]. A common
computation tree for elements in the center part ce. A list of computation trees forelements on the right edge [r1 ]. Elements involved in the computation are boxed bydashed lines.


+

+
+

 c−2 × c−1 × c0 × c1 ×
b l0

b l1



+

−
→
u[0]

−
→
u[1]

+

+


c−2 × c−1 × c0 × c1 × 
b l1

−
→
u[0]

−
→
u[1]

−
→
u[2]



+
+

+

c−2 × c−1 × c0 ×

c1 ×

u≪2 u≪1

u≫1

u



+

+
+


 c−2 × c−1 × c0 × c1 × 
←
−
u[2]

←
−
u[1]

←
−
u[0]

br

Figure 5.2. The triple describing computation of next.
center part, and rs is the list of computation trees for the right edge. Computation
trees in normal forms are defined as follows.
data Tree α = Node ((β, γ) → α) (Tree β) (Tree γ)| Leafv (β → α) (Var β)| Leafc αdata Var α = Var [α] Int| Fix [α] Int Direction| Holedata Direction = FromL | FromR
A node of the tree holds the left and right children zipped by zip, and a composed
function applied by successive maps. There are two kinds of leaves: Leafc denotes
a constant introduced by shift≪ or shift≫, and Leafv denotes input lists and holds acomposed function applied by successive maps. The data structure Var α denotes
access to the input lists: Var represents index access of lists in the center part;and Fix represents fixed-index access in computations of edge elements. Therefore,
Var holds the list and the amount of shifting, while Fix holds the following three:80

5. Domain-Specific Optimization for Skeleton Programs

the input list to be accessed, the index, and the origin of the index. The originof the fixed index is specified by Direction. A special value Hole will be used informalization of parallel implementation, which indicates that an element of otherprocessor will fill out the hole to complete the tree.
−→
For example, expression add (c−2 ×bl1 , c−1 ×u[0]) is described as follows.
Node add (Leafc (c−2 ×bl1 )) (Leafv (c−1 ×) (Fix u 0 FromL))
A part c−2 ×u≪2 of common tree is described as follows.
Leafv (c−2 ×) (Var u (−2))
Note that negative value of the amount of shifting means shifting to the left.
Semantics of Normal Form
We will give a semantics of the normal form by sequential evaluation of the normal
form. In the evaluation, elements in the center part are computed by a single loopwith the common computation tree, and each element on both edges is computedby its own computation tree.
First, we define the evaluation of computation trees as evaluation function evalT .
evalT :: Tree α → Int → α
evalT (Node f l r) i = f (evalT l i, evalT r i)
evalT (Leafv f v) i = f (evalV v i)
evalT (Leafc c) i
= c
The evaluation function evalT performs computation according to the definition ofcomputation trees, applying functions stored in the trees to the elements of the
input lists along with the tree structures. It flows the index of the element being
computed through the evaluation to the other evaluation function evalV . The auxiliary function evalV processes index accessing of input lists. The result of evalV for
Hole is not defined.
evalV :: Var α → Int → α
evalV (Var u s) i
= at u (i − s)evalV (Fix u s FromL)= at u s= at (reverse u) sevalV (Fix u s FromR)
Here, function at returns the element at the given index, and defined as follows.
at i (a : x) = if i = 0 then a else at (i − 1) x
Next, we will define the evaluation function evalT0 for elements on the left and rightedges. The evaluation is defined by evalT ignoring the index.
evalT0 :: Tree α → α
evalT0 x = evalT x 0

5.2. Computation Involving a Finite Number of Neighbor Elements

81

Then, using the above evaluation functions, we define a sequential program evalthat evaluates the normal form. Here, the lengths of involved lists are supposed tobe n.
eval :: NForm α → [α]eval [[ ls, zms, rs ]] = map evalT0 ls ++ map (evalT zms) idces ++ map evalT0 rs
where l = length ls ; r = length rs
idces = [l..(n − r − 1)]
Each element on both edges is calculated by its own computation tree using evalT0
defined above. The center part is calculated by a single loop (map (evalT zms)) withthe common computation tree.

5.2.3

Fusion Rules for Transformation to a Normal Form

The second step of our strategy is to define fusion rules to transform skeleton compositions to normal forms. These rules should be able to transform any of the target
skeleton compositions to normal forms. In this section, we give fusion rules to transform a skeleton program Program (see Section 5.2.1) into the normal form definedso far.
We first give formal definition of the fusion rules. Then, we will show example
transformations with the fusion rules.
Transformation with Fusion Rules
We will define function compile that transforms a skeleton program into the normal
form by one-by-one application of fusion rules.
compile :: Program α → NForm α
compile (map f x) = fuseMap f (compile x)compile (shift≪ e x) = fuseShift ≪ e (compile x)compile (shift≫ e x) = fuseShift ≫ e (compile x)compile (zip x y) = fuseZip (compile x) (compile y)compile (x)= [[ [ ], Leafv id (Var x 0), [ ] ]]Fusion rules for skeletons are defined as functions fuseMap, fuseShift ≪, fuseShift ≫,and fuseZip, which will be define below. Figure 5.3 illustrates the fusion rules.
Fusion Rule for map
Fusion of skeleton map is performed by composing the given function to roots ofcomputation trees.
fuseMap :: (α → β) → NForm α → NForm β
fuseMap f [[ ls, zms, rs ]] = [[ map (comp f ) ls, comp f zms, map (comp f ) rs ]]82

5. Domain-Specific Optimization for Skeleton Programs

map f

shift≫ e⇓(a) Fusion of map composes the given function toeach tree of the normal form. The structure of
the normal form is not changed.

⇓(b) Fusion of shift≫ discards the rightmost computation tree and introduces a new constant tree
to the leftmost. The amount of shifting in thetree for center part is updated by 1.

zip
shift≪ e⇓⇓(c) Fusion of shift≪ discards the leftmost computation tree and introduces a new constant tree
to the rightmost. The amount of shifting in thetree for center part is updated by −1.

(d) Fusion of zip instantiates the shaded parts sothat the sizes of lists of trees for edges become
uniform, and zips the corresponding trees.

Figure 5.3. An image of fusion rules. Rectangles represent lists. The three parts
separated by vertical lines correspond to the triple of the normal form. Changed
parts in the resulting normal form are shaded.
Composition of a function is given by the following comp.
comp :: (α → β) → Tree α → Tree βcomp f (Node g l r) = Node (f ◦ g) l r
comp f (Leafv g v) = Leafv (f ◦ g) v
comp f (Leafc c)= Leafc (f c)
For non-constant roots, comp composes the given function to the function held in
the root. For constant roots (i.e. constant leaves), comp applies the given function
to the constant to generate a new constant root.
Fusion Rules for shift≪ and shift≫
Fusion of skeletons shift≪ and shift≫ is performed by insertion and deletion of theleftmost and the rightmost trees, and update of the amount of shifting.
fuseShift ≪ :: α → NForm α → NForm α
fuseShift ≪ e [[ ls, zms, rs ]] = [[ tail ls, slide (−1) zms, rs ++ [Leafc e] ]]fuseShift ≫ :: α → NForm α → NForm α
fuseShift ≫ e [[ ls, zms, rs ]] = [[ [Leafc e] ++ ls, slide 1 zms, init rs ]]For the left-shift, the fusion discards the leftmost computation tree, and introduces
the constant computation tree of the given constant e to the rightmost. Then, it5.2. Computation Involving a Finite Number of Neighbor Elements

83

updates the amount of shifting in the common computation tree for the center part.
This update is performed by the following slide.
slide :: Int → Tree α → Tree αslide d (Node f l r) = Node f (slide d l) (slide d r)
slide d (Leafv f (Var x s)) = Leafv f (Var x (s + d))
slide d x = x
The right-shift is similar to the left-shift.
Fusion Rule for zip
Fusion of skeleton zip needs unification of the lengths of lists of computation trees
for edges. Thus, the common trees for the center parts are instantiated to expand
the lists of edge computation trees. Then, each pair of corresponding trees is zippedby introducing a new node with id function.
fuseZip :: NForm α → NForm β → NForm (α, β)fuseZip [[ ls 1 , zms 1 , rs 1 ]] [[ ls 2 , zms 2 , rs2 ]]= let zms = Node id zms 1 zms 2
ls = trim FromL ls 1 ls 2 zms 1 zms 2
rs = trim FromR (reverse rs 1 ) (reverse rs 2 ) zms 1 zms 2
in [[ ls, zms, reverse rs ]]The function trim defined below unifies the lengths of lists by instantiating common
tree zms k by a function insts defined below.
trim :: Direction → [Tree α] → [Tree β] → Tree α → Tree β → [Tree (α, β)]trim d ts 1 ts 2 zms 1 zms 2= let n1 = length ts 1 ; n2 = length ts 2
(ts ′1 , ts ′2 ) = (ts 1 ++ insts d zms 1 n1 n2 , ts 2 ++ insts d zms 2 n2 n1 )in zipwith (Node id ) ts ′1 ts ′2
The instantiation is performed by the following insts and inst.
insts :: Direction → Tree α → Int → Int → [Tree α]
insts d zms s e = map (inst d zms) [s..(e − 1)]inst d (Node f l r) i = Node f (inst d l i) (inst d r i)
inst d (Leafv f (Var x s)) i = let s′ = case d of FromL → s; FromR → −sin Leafv f (Fix x (−s′ + i) d)
inst d x i = x
Completeness of the Fusion Rules
These four fusion rules and the base case rule can transform any skeleton program
defined by Program into the normal form. We conclude this fact as a theorem.

84

5. Domain-Specific Optimization for Skeleton Programs

Theorem 5.1. Any skeleton program defined by Program can be transformed into
the normal form by using the four fusion rules and the base case rule, while keeping
the result of the whole computation. That is, the following equation holds for any
prog of Program.
evalP prog = eval (compile prog)
Proof. This is proved by induction on the structure of Program. The base case isshown by the transformation of an input list. Induction cases are shown by the four
fusion rules.
Complete proof is shown in Appendix B.
This completeness of fusion rules is the sharp contrast of our domain-specificfusions to general fusions, which do not determine how widely they are applicable.
Example Transformation
As a brief explanation of the rules, we transform the example next into the normal
form. For readability, we use a brief notation used in the previous examples insteadof the actual data structures defined so far. In the rest of this section, we will use ⇒to show the transformation; the left hand side is the structure being transformed,
and the right hand side is the resulting normal form.
The most simplest case is the transformation of the argument list u. List u needs→−
only the common computation tree u [i] that is just the element of u.
u ⇒ [[ [ ], id u, [ ] ]]Here, id u is the brief notation of Leafv id (Var u 0).
Next, we transform shift≫ bl1 u. This shift≫ introduces the constant bl1 to theleftmost element. Thus, a new computation tree of the constant bl1 is introduced to
the normal form.
shift≫ bl1 [[ [ ], id u, [ ] ]] ⇒ [[ [bl1 ], id u≫1 , [ ] ]]Also, the amount of shifting in the common tree is updated by 1.
Then, we fuse map (c−1 ×) to the above result.
map (c−1 ×) [[ [bl1 ], id u≫1 , [ ] ]] ⇒ [[ [c−1 ×bl1 ], c−1 ×u≫1 , [ ] ]]The constant bl1 is replaced by c−1 × bl1 , and the function (c−1 ×) is composed to id
held in the root of the common tree. Since id is the identity of function compositions,
id is removed in the result.
Similarly, other applications of shift≫, shift≪ and map result in the following
normal forms.
map (c0 ×) u ⇒ [[ [ ], c0 × u, [ ] ]]map (c1 ×) (shift≪ br u) ⇒ [[ [ ], c1 × u≪1 , [c1 × br ] ]]map (c−2 ×) (shift≫ bl0 (shift≫ bl1 u)) ⇒ [[ [c−2 × bl0 , c−2 × bl1 ], c−2 × u≫2 , [ ] ]]

5.2. Computation Involving a Finite Number of Neighbor Elements

85

In the second transformation, a new tree is introduced by shift≪ to the rightmost,and the amount of shifting in the common tree is updated by 1 to the left. In thelast transformation, there are two computation trees for elements on the left edge,
and the amount of shifting in the common tree becomes 2. These are introduced by
two shift≫s.
Next, we perform fusion of zip to transform zip v0′ v1′ , i.e.
zip [[ [ ], c0 × u, [ ] ]] [[ [ ], c1 × u≪1 , [c1 × br ] ]].
Since the lengths of lists of two normal forms to be zipped are not the same (i.e.
[ ] and [c1 × br ] for the right edges), we have to unify the lengths by instantiating
the common trees. Instantiation means fixing the indices in the common trees forelements on the edges. The result of instantiation and zip is as follows.
←−[[ [ ], (c0 × u, c1 × u≪1 ), [(c0 × u[0], c1 × br )] ]]Here, instantiation of the common tree of the first normal form c0 × u results in←−
c0 × u[0], and it is zipped with the rightmost tree of the second normal form to
make the new rightmost tree.
′′
Similarly, we obtain the following normal form from zip v−2
v−1
.
−
→
[[ [(c−2 × bl0 , c−1 × bl1 ), (c−2 × bl1 , c−1 × u[0])], (c−2 × u≫2 , c−1 × u≫1 ), [ ] ]]Continuing these fusions, we finally obtain the following normal form for theexample next.
−→−→[[ [ add (add (c−2 × bl0 , c−1 × bl1 ), add (c0 × u[0], c1 × u[1])),
−
→
−
→
−
→
add (add (c−2 × bl1 , c−1 × u[0]), add (c0 × u[1], c1 × u[2]))],add (add (c−2 × u≪2 , c−1 × u≪1 ), add (c0 × u, c1 × u≫1 )),
←
−
←
−
←
−
[add (add (c−2 × u[2], c−1 × u[1]), add (c0 × u[0], c1 × br ))] ]]5.2.4

Parallel Implementation of Normal Form

The third step of our strategy is to design parallel implementation of the normal
form. In this section, we explain the parallel implementation of the normal form.
The implementation of the normal form is not necessarily a skeleton composition.
Based on parallel implementation of existing skeletons [MIEH06], we consider
parallel implementation of the normal form consisting of four steps: (1) distributionof a normal form (input lists), (2) the first local computation, (3) global communication, (4) the second local computation. We explain the idea of parallel implementation using the example next. Note that gathering and redistribution of theresult of the computation may be canceled when the result will be used as the inputof the next computation. Figure 5.4 shows an image of parallel implementation of

86

5. Domain-Specific Optimization for Skeleton Programs

ւ

(1)
(2)

ց

dist

⇓ eval ′⇓ eval ′

(3)

ց

ւ

(4)

ւ ց

combine(1) Input lists used in a normal form are divided into two parts to generate two local normal forms
for two processors. Then, these normal forms are distributed to processors.
(2) Each processor calculates elements that can be calculated by local data. Elements on the edges
cannot be calculated because they need data on other processors.
(3) To calculate the elements on the edges, communication is performed to complete the data.
(4) We perform calculation of the elements on the edges to complete the computation.

Figure 5.4. An image of parallel implementation of the normal form (two processors).
the normal form using two processors. We will explain the idea with two processors
here, and will formalize its generalization later.
First, we distribute the normal form among processors. The input list u is divided
into two parts: u = u1 ++ u2 . There are two processors, and each processor has apart of the divided list. Note that u may already be distributed and the distribution
phase may be skipped when u is the result of another computation of a normal
form. Letting the original normal form be [[ ls, zms, rs ]], each processor has one of
the following distributed normal forms.
nf 1 = [[ ls, zms 1 , [ ] ]] ; nf 2 = [[ [ ], zms 2 , rs ]]Here, the list of computation trees for the left edge is held in the first normal form
nf 1 , while the list of computation trees for the right edge is held in the last normal
form nf 2 . The common computation tree zms k of the normal form nf k is createdfrom zms by replacing the input list u with the part of the list uk . Thus, for theexample next, the distributed normal form nf k is as follows.
add (add (c−2 ×uk ≪2 , c−1 ×uk ≪1 ), add (c0 ×uk , c1 ×uk ≫1 ))Next, in the first local step, processors calculate own partial results in parallel.
Since elements on the edges of the distributed normal forms needs elements of both
u1 and u2 , these elements cannot be calculated in this phase. For example, therightmost element calculated by nf 1 needs an element of u2 , which is underlined in
the following expression.
−
→
←
−
←
−
←
−
add (add (c−2 ×u1 [2], c−1 ×u1 [1]), add (c0 ×u1 [0], c1 ×u2 [0]))Since these elements cannot be calculated in this local phase, computation trees forthese elements are held until the global computation phase. Here, we will put holes

5.2. Computation Involving a Finite Number of Neighbor Elements87• to the places of elements that are not available in this phase.
←
−
←
−
←
−
add (add (c−2 ×u1 [2], c−1 ×u1 [1]), add (c0 ×u1 [0], c1 ×•))Similarly, the left edge of nf 2 generates the following computation tree with holes.
−→
add (add (c−2 ×•, c−1 ×•), add (c0 ×•, c1 ×u2 [0]))Note that this tree complement the previous tree generated from nf 1 ; merging them
we have the complete tree. Generally, letting the maximum amounts of shifting
to the left and the right be l and r, the number of computation trees with holes
generated on the break of division is l + r. Trees on both sides of the break arecomplementary to each other.
Third, in the global communication step, neighboring processors communicate
incomplete trees with holes to each other to complete the trees. The first local
computation can hide the time of this communication phase.
Fourth, in the second local step, each processor calculates the elements on its
edges with the completed trees to finish the computation.
In our example program next, the first normal form nf 1 generates the following
trees with holes.
←
−
←
−
←
−
add (add (c−2 ×u1 [2], c−1 ×u1 [1]), add (c0 ×u1 [0], c1 ×•))←−←−add (add (c−2 ×u1 [1], c−1 ×u1 [0]), add (c0 ×•, c1 ×•))←−add (add (c−2 ×u1 [0], c−1 ×•), add (c0 ×•, c1 ×•))The second normal form nf 2 generates the following trees.
−→add (add (c−2 ×•, c−1 ×•), add (c0 ×•, c1 ×u2 [0]))−→−→add (add (c−2 ×•, c−1 ×•), add (c0 ×u2 [0], c1 ×u2 [1]))
−
→
−
→
−
→
add (add (c−2 ×•, c−1 ×u2 [0]), add (c0 ×u2 [1], c1 ×u2 [2]))Zipping these trees, we obtain the following complete computation trees for elementson the break.
−
→
←
−
←
−
←
−
add (add (c−2 ×u1 [2], c−1 ×u1 [1]), add (c0 ×u1 [0], c1 ×u2 [0]))
−
→
−
→
←
−
←
−
add (add (c−2 ×u1 [1], c−1 ×u1 [0]), add (c0 ×u2 [0], c1 ×u2 [1]))
−
→
−
→
−
→
←
−
add (add (c−2 ×u1 [0], c−1 ×u2 [0]), add (c0 ×u2 [1], c1 ×u2 [2]))After these four steps, these completed results are gathered to the root processor, or
become a new input to another normal form. Distribution of input will be skippedin the latter case.
Summarizing the above ideas, we get the following general structure of parallel
implementation of the normal form.
parEval = globalReduction ◦ map (localEval ) ◦ dist

88

5. Domain-Specific Optimization for Skeleton Programs

First, a normal form is distributed among processors by dist (step (1)). Next, eachprocessor evaluates a part of distributed normal forms by localEval (step (2)). Then,these results are combined globally by globalReduction (step (3) and (4)). Local
computation localEval takes a distributed normal form, and generates a triple of its
partial result (pls, cs, prs), where cs is the calculated elements on the center part,and pls and prs are computation trees with holes for elements on left and right edges.
Global computation globalReduction communicates partial results (pls 1 , cs 1 , prs 1 )and (pls 2 , cs 2 , prs 2 ) of both sides of an edge, then generates a new partial result(pls 1 , cs 1 ++ glue prs 1 pls 2 ++ cs 2 , prs 2 ). Here, glue denotes the complement process.
Then, the final result is obtained by extracting the center result of the final partial
result.
We will complete the definition of the general computation in the next section.
Formalization of Parallel Implementation of Normal Form
We formalize the parallel implementation explained in the previous section. In therest of this section, p means the number of processors and n means the length ofthe lists involved in the computation.
First, we define the distribution dist that divides a normal form into p parts.
dist :: Int → NForm α → [NForm α]
dist p [[ ls, zms, rs ]] = let divs = division p nzmss = distribute zms divs
lss = ls : dupl (p − 1) [ ]rss = dupl (p − 1) [ ] ++ [rs]in zip3 lss zmss rss
Here, function dupl generates a list of the given element of the given length, anddefined as follows.
dupl 0 a = [ ]dupl n a = [a] ++ dupl (n − 1) a

Function division p n calculates the division of input lists, and distribute zms divs
distributes the input lists according to the division. This process generates distributes a computation tree zms i that is generated by replacing a list u (= u1 ++· · · ++ ui ++ · · · ++ up ) in the original tree zms with ui , and zms i is distributed to ith
processor and held in the normal form nf i (this is generated by zip3 lss zmss rss).
The list of computation trees for the left edge is held in the first normal form nf 1 ,while the list of computation trees for the right edge is held in the last normal form
nf p .
Next, we will define the first local computation localEval . To this end, we define
the triple of the partial result PResult of the local computation, and define some
useful functions.
type PResult α β = ([Tree α], β, [Tree α])5.2. Computation Involving a Finite Number of Neighbor Elements

89

We use a special brackets hh pls, cs, prs ii to denote a partial result. Here, cs isthe computed elements on the center part, and pls and prs are computation trees
with holes for elements on left and right edges. We abstract the type of calculated
elements on the center part as β for generality.
An auxiliary function maxShift is defined as follows to calculate the maximum
amount of shifting.
maxShift :: Tree α → (Int, Int)
maxShift (Node l r) = let (l1 , r1 ) = maxShift l(l2 , r2 ) = maxShift rin (max l1 l2 , max r1 r2 )maxShift (Leafv (Var s)) = if s < 0 then (−s, 0) else (0, s)
maxShift= (0, 0)We assume that the maximum amount of shifting is less than n/p.
A general function gEval ′ to perform the local computation is defined as follows.
gEval ′ :: (NForm α → [Int] → β) → NForm α → PResult α β
gEval ′ fc [[ ls, zms, rs ]] = hh pls, cs, prs ii
whereidces = [r..(n − l − 1)](l, r) = maxShift zms
cs = fc (ls, zms, rs) idces
pls = map (instP FromL zms) [(−l)..(r − 1)]
prs = map (instP FromR zms) (reverse [(−r)..(l − 1)])This gEval ′ takes a function to calculate the center part of the partial result, andgenerates a partial result consisting of the result computed by the given function
and two lists of computation trees with holes, which are generated by the following
instantiation function instP .
instP :: Direction → Tree α → Int → Tree α
instP d (Node f l r) i = Node f (instP d l i) (instP d r i)
instP d (Leafv f (Var x s)) i = let s′ = case d of FromL → s; FromR → −si′ = (−s′ + i)
in if (i′ ≤ (length x))||(i′ < 0) then Leafv f Holeelse Leafv f (Fix x i′ d)instP d x i = x
This instP is the same of inst except that it substitutes a hole for an element thatis not available in this phase.
Now, we define the function localEval that performs local computation of the

90

5. Domain-Specific Optimization for Skeleton Programs

normal form as follows using gEval ′ .
localEval :: NFormα → PResult α [α]localEval = gEval ′ fcMap
where fcMap [[ ls, zms, rs ]] idces= map evalT0 ls ++ map (evalT zms) idces ++ map evalT0 rs
The defined function fcMap evaluates the computation tree zms against indices given
by gEval ′ , and returns a list of the resulting elements.
Next, we will define globalReduction: the final global communication and completion of partial computation trees. To this end, we define the process of completionof a computation tree from two trees complementary to each other.
combine :: Tree α → Tree α → Tree α
combine (Node f1 l1 r1 ) (Node f2 l2 r2 ) =combine (Leafv f1 Hole) (Leafv f2 v)=
combine (Leafv f1 v) (Leafv f2 Hole)) =combine (Leafc c) (Leafc c′ )=Node f1 (combine l1 l2 ) (combine r1 r2 )Leafv f2 v
Leafv f1 v
Leafc cThis function completes the tree by filling holes with values held in the other tree.
Using this auxiliary function, we define an operator ⊞(·,·) for global reduction.
(⊞(·,·) ) :: (α → α → α) → (β → α) → PResult β α → PResult β α → PResult β αhh pls 1 , cs 1 , prs 1 ii ⊞(⊕,f ) hh pls 2 , cs 2 , prs 2 ii = hh pls 1 , cs, prs 2 ii
where es = zipwith ((evalT0 ◦) ◦ combine) prs 1 pls 2
cs = cs 1 ⊕ cata (⊕) f es ⊕ cs 2
Here, es is the list of elements computed from the completed trees by evaluating thetrees with evalT0 . The defined operator takes an associative operator and a function
to compute the reduction of the elements on breaks computed from the completed
trees. This reduction is done by the following general function.
cata :: (α → α → α) → (β → α) → [β] → αcata (⊕) f [ ]= ı⊕cata (⊕) f (a : x) = f a ⊕ cata (⊕) f x
Especially, cata (++) [·] = id , and the operator ⊞(++,[·]) simply returns the resulting
list.
Using the functions defined above, we define the global computation globalReduction.
globalReduction :: [PResult α [α]] → [α]globalReduction = extract ◦ reduce (⊞(++,[·]) )The last function extract extracts the center value from the final partial result.
extract :: PResult α β → β
extract hh pls, cs, prs ii = cs

5.2. Computation Involving a Finite Number of Neighbor Elements

91

Now, we have the following completed definition of parEval .
parEval :: Int → NForm α → [α]parEval p = globalReduction ◦ map localEval ◦ dist p

5.2.5

Expansion of Target Programs

Based on the results on Program, we expand our target programs to ProgramS , which
includes accumulation by scan′ or scanr′ , and ProgramR , which includes reduction
by reduce (see Section 5.2.1).
Target Programs with Accumulation
First, we expand our target programs to ProgramS . A target skeleton program of
ProgramS has one accumulation by scan′ or scanr′ after the computation of Programthat involves neighbor elements using combination of shift≪, shift≫, zip and map. The
target skeleton program also has arbitrary number of map after the accumulation.
As an example of the target programs, consider a program to solve a tridiagonallinear system of equations. The output of the program is xs = [x1 , . . . , xn ] thatsatisfies the following linear equations for the given coefficients ds = [d1 , . . . , dn ],es = [e1 , . . . , en ], fs = [f1 , . . . , fn ], bs = [b1 , . . . , bn ].
 




x1
d 1 f1b1
  x2  e 2 d 2 f2
 b2 
 










x
e
d
f
b
3
3
3
  2 
 2 
  ..  =  .. ... ...
...
  . 

 . 
 








bn−1 en−1 dn−1 fn−1
xn−1
en
dn
xn
bn
|

{z
A

} | {z }| {z }x=b

A skeleton program that solves this tridiagonal linear system of equations is given
in Figure 5.5. This program is based on the LU decomposition of the coefficientmatrix A = LU [Sto73]. Here, us corresponds to the upper triangular matrix U ,ms corresponds to the lower triangular matrix L, ys is the solution of the linear
equation Lys = bs, and xs the solution of the linear equation Axs = bs. The
operators ⊗1 , ⊗2 and ⊗3 are multiplication of 2 × 2 matrices, although ⊗2 and ⊗3
omit the half of the elements. This program consists of three parts: the first part
performs the LU decomposition with scan′ to obtain us and ms, the second part
performs the forward substitution by scan′ to obtain ys, and the third part performs
the backward substitution by scanr′ to obtain xs.
Now, we will extend the normal form of Program to hold the following three:the direction, the operator, the initial value of the accumulation, and the composed

92

5. Domain-Specific Optimization for Skeleton Programs

solveTS ds es fs bs= let us = map g2 (scan′ (⊗1 ) eye (map g1 (zip ds (map mul (zip es (shift≫ 0 fs))))))
ms = map div (zip es (shift≫ ∞ us))ys = map g5 (scan′ (⊗2 ) eye ′ (map g3 (zip ms bs)))xs = map g5 (scanr′ (⊗3 ) eye ′ (map g4 (zip us (zip ys fs))))in xs
where (a11 , a12 , a21 , a22 ) ⊗1 (b11 , b12 , b21 , b22 )= (a11 ∗ b11 + a12 ∗ b21 , a11 ∗ b12 + a12 ∗ b22 ,a21 ∗ b11 + a22 ∗ b21 , a21 ∗ b12 + a22 ∗ b22 )(a11 , a21 ) ⊗2 (b11 , b21 ) = (a11 ∗ b11 , a21 ∗ b11 + b21 )(a11 , a12 ) ⊗3 (b11 , b12 ) = (a11 ∗ b11 , a11 ∗ b12 + a12 )
g1 (d, ef ) = (d, 1, −ef , 0)
g2 (a11 , a12 , a21 , a22 ) = a11 /a12
g3 (m, b) = (−m, b)
g4 (u, (y, f )) = (−f /u, y/u)g5 (a11 , a12 ) = a12
mul a b = a ∗ b
div a b = a/beye = (1, 0, 0, 1)
eye ′ = (1, 0)

Figure 5.5. A skeleton program that solves the tridiagonal linear system of equations.
function applied by the last maps. The extended normal form NFormS is defined
as follows.
type NFormS α = (NForm β, (Direction, β → β → β, β, β → α))
In the extended normal form ([[ ls, zms, rs ]], (d, ⊕, e, f )), [[ ls, zms, rs ]] specifies thecomputation before the accumulation, d is the direction of accumulation, ⊕ is theassociative binary operator used in the accumulation, e is the initial element of theaccumulation, and f is a function applied to each element after the accumulation.
The direction d is FromL for the accumulation by scan′ , and FromR for scanr′ .
For example, the example solveTS is described by the following three normal
forms.
us ⇒ ([[ [g1 (ds , mul (es , 0))], g1 (ds , mul (es , fs ≫1 )), [ ] ]], (FromL, ⊗1 , eye, g2 ))ys ⇒ ([[ [g3 (div (es , ∞), bs )], g3 (div (es , us ≫1 ), bs ), [ ] ]], (FromL, ⊗2 , eye ′ , g5 ))xs ⇒ ([[ [ ], g4 (us , (ys , fs )), [ ] ]], (FromR, ⊗3 , eye ′ , g5 ))Note that the computation of ms is absorbed by the computation of ys.
Next, we will extend the fusion rules to handle scan′ , scanr′ and map after accumulation. The transformation of a skeleton program into an extended normal form

5.2. Computation Involving a Finite Number of Neighbor Elements

93

is performed by the following function compileS with fusion rules one by one.
compileS :: ProgramS α → NFormS α
compileS (scan (⊕) e x) = fuseScan (⊕) e (compile x)compileS (scanr (⊕) e x) = fuseScanr (⊕) e (compile x)compileS (map f x)
= fuseMapS f (compileS x)
Transformation of the computation before the accumulation is done by compile
defined in Section 5.2.3.
Fusion rules for accumulations are as follows.
fuseScanr :: (α → α → α) → α → NForm α → NFormS α
fuseScanr (⊕) e [[ ls, zms, rs ]] = ([[ ls, zms, rs ]], (FromR, (⊕), e, id ))fuseScan :: (α → α → α) → α → NForm α → NFormS α
fuseScan (⊕) e [[ ls, zms, rs ]] = ([[ ls, zms, rs ]], (FromL, (⊕), e, id ))Each rule appends the operator ⊕ and the initial element e to the normal form, andmarks the direction of the accumulation by FromL or FromR. The last element of
the extended normal form is initialized by the identity function id .
The fusion rule for map after accumulation is as follows.
fuseMapS :: (β → α) → NFormS β → NFormS α
fuseMapS f = (nf , (d, (⊕), e, g)) = (nf , (d, (⊕), e, f ◦ g))
This rule merely composes the given function f to the function g in the extended
normal form.
It is obvious that any target program can be transformed into an extended normal
form by the above fusion rules and the fusion rules defined in Section 5.2.3.
Finally, we will extend the parallel implementation of the normal form. Basedon parallel implementation of skeleton scan′ , we design parallel implementation of
the normal form. Parallel implementation of skeleton scan′ is given as follows.
scan′ (⊕) e = reduce (++) ◦ zipwithP (afterScan (⊕))◦ ((prescan (⊕) e ◦ map last)△id )◦ map (scan′ (⊕) ı⊕ ) ◦ dist p
where afterScan (⊕) a x = map (a⊕) xHere, zipwithP f (x, y) = zipwith f x y, and (f △g) x = (f x, g x). First, this implementation performs local accumulation in parallel by scan′ . Then, it performs global
accumulation by prescan defined below. Finally, afterScan adds the accumulated
value calculated by prescan to each element of the result of the local accumulation.
prescan :: (α → α → α) → α → [α] → [α]prescan (⊕) e [ ]= []prescan (⊕) e (a : x) = e : prescan (⊕) (e ⊕ a) x

94

5. Domain-Specific Optimization for Skeleton Programs

Based on the parallel implementation of scan′ , we consider the following parallel
implementation of a normal form with accumulation. We only show the implementation of the normal form with accumulation by scan′ . The implementation foraccumulation by scanr′ is similar to that by scan′ .
parEvalScan :: Int→NFormS α→[α]parEvalScan p (nf , (FromL, ⊕, e, f ))= (reduce (++) ◦ zipwithP (evalAfterScan (⊕) f )◦ ((prescan (⊞(⊕,id) ) ([ ], e, [ ]) ◦ map takeLast)△id )◦ map (evalScan ′ (⊕) ı⊕ ) ◦ dist p) nf
Here, takeLast hh pls, cs, prs ii = hh pls, last cs, prs ii. Basic structure is the same as
the implementation of scan′ . Main difference is that local and global computations
deal with triples of partial results defined in the previous section. The local computation evalScan ′ that generates a partial result is defined as follows. The generated
partial result holds the result of local accumulation in the center of the triple.
evalScan ′ :: (α → α → α) → α → NForm α → PResult α [α]evalScan ′ (⊕) e = gEval ′ fcScan
where fcScan [[ ls, zms, rs ]] idces = sls ++ scs ++ srs
where sls = scata (⊕) e evalT0 ls
e′ = last (e : sls)
scs = scata (⊕) e′ (evalT zms) idcese′′ = last (e′ : scs)
srs = scata (⊕) e′′ evalT0 rs
Here, scata defined below performs accumulation and evaluation of the computation
tree at the same time.
scata :: (α → α → α) → α → (β → α) → [β] → [α]scata (⊕) e f [ ]= []scata (⊕) e f (a : x) = let e′ = (e ⊕ f a)in e′ : scata (⊕) e′ f x
The final local computation evalAfterScan (⊕) calculates accumulation of elementson the edges and adds the accumulated value to the result of the local computation.
evalAfterScan :: (α → α → α) → (α → β) → PResult α α → PResult α [α] → [β]evalAfterScan (⊕) f hh epls, e, eprs ii hh pls, cs, prs ii = res
where fe (epr , pl ) = evalT0 (combine epr pl ))sls = scata (⊕) e fe (zip eprs pls)
e′ = last (e : sls)
scs = map (f ◦ e′ ⊕) cs
res = map f sls ++ scs

5.2. Computation Involving a Finite Number of Neighbor Elements

95

Target Programs with Reduction
Next, we expand our target programs to ProgramR .
We will extend the normal form of Program to hold the operator of the reduction.
The extended normal form NFormR is defined as follows.
type NFormR α = (NForm β, β → β → β)In the extended normal form ([[ ls, zms, rs ]], ⊕), [[ ls, zms, rs ]] specifies the computation before the accumulation, and ⊕ is the associative binary operator used in the
reduction.
Next, we will extend the fusion rules to handle reduce. The transformation ofa skeleton program into an extended normal form is performed by the following
function compileR and fusion rules one by one.
compileS :: ProgramS α → NFormS α
compileR (reduce (⊕) x) = fuseReduce (⊕) (compile x)
Transformation of the computation before the reduction is done by compile definedin Section 5.2.3.
The fusion rule for reduction is as follows.
fuseReduce :: (α → α → α) → NForm α → NFormR α
fuseReduce (⊕) [[ ls, zms, rs ]] = ([[ ls, zms, rs ]], (⊕))
The rule appends the operator ⊕ to the normal form.
It is obvious that any target program can be transformed into an extended normal
form by the above fusion rule and the fusion rules defined in Section 5.2.3.
Now, we will give parallel implementation for ProgramR that perform reduction
by reduce to the result of a normal form. These programs can be executed in parallelby performing reduction, instead of generation of a list, in the implementation of anormal form. That is, parallel implementation is obtained by giving the reduction
operator to gEval ′ and ⊞(·,·) .
The function evalReduce ′ that performs local reduction is defined as follows.
evalReduce ′ :: (α → α → α) → NForm α → PResult α αevalReduce ′ (⊕) = gEval ′ fcReduce
where fcReduce [[ ls, zms, rs ]] idces= cata (⊕) evalT0 ls ⊕ cata (⊕) (evalT zms) idces ⊕ cata (⊕) evalT0 rs
This function performs reduction and evaluation of computation trees at the sametime by cata (⊕) evalT0 and cata (⊕) (evalT zms). Using this function, parallel
implementation of a normal form with reduction is defined as follows.
parEvalReduce :: Int → NFormR α → α
parEvalReduce p (nf , ⊕)= (extract ◦ reduce (⊞(⊕,id) ) ◦ map (evalReduce ′ (⊕)) ◦ dist p) nf

96

5. Domain-Specific Optimization for Skeleton Programs

5.3

Nested Reductions: Involving an Infinite Number of Neighbor Elements

In this section, we will focus on computations of nested reductions: nested data
structures are generated from flat structures, and they are consumed by two-level
nested reductions. This is a generalized version of computation discussed in theprevious section, in the sense that the computation involving a finite number ofneighbor elements can be specified as nested reductions.
First, we will introduce the general form of nested reductions. Then, we will introduce some useful functions to generate nested structures, and show some example
concrete problems. Finally, we will develop shortcuts of fusion optimizations, i.e.,theorems to dispatch efficient implementations to general forms that satisfy somespecific conditions.

5.3.1

Generate-and-test Specifications: General Forms of
Nested Reductions

We will introduce general forms “generate-and-test specifications” of nested reductions on lists and two-dimensional arrays.
First, we will introduce the general form for nested reductions on lists.
Definition 5.2 (Generate-and-test specification for lists). The general form fornested reductions on lists is given as follows.
([f, ⊕]) ◦ map ([g, ⊗]) ◦ filter p ◦gg
Here, gg is a function to generate nested lists. The generated lists are filtered withpredicate p. Then, the filtered lists are consumed by the nested reductions: ([f, ⊕])for the outer list and ([g, ⊗]) for the inner lists.
Note that homomorphism ([f, ⊕]) is equivalent to the composition reduce (⊕) ◦ map f .
Especially, simple nested reductions with two binary operators ⊕ and ⊗ for the generation by gg can be described in the generate-and-test specification as follows.
reduce (⊕) ◦ map (reduce (⊗)) ◦gg = ([id , ⊕]) ◦ map ([id , ⊗]) ◦ filter true ◦gg
Here, true is the predicate returning always True. The equality of the left handside (the simple nested reductions) and the right hand side (the generate-andtest specification) is easily shown. First, by the definition of reduce (⊕), we have([id , ⊕]) = reduce (⊕) and ([id , ⊗]) = reduce (⊗). Next, by the definition of filter p,filter true = id since true returns always True.
It is worth noting that the identity function id can be described with homomorphism, i.e., id = ([[·], ++]), and the outer reduction may be omitted when it is theidentity function.
Next, we will introduce the general form for two-dimensional arrays.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

97

Definition 5.3 (Generate-and-test specification for two-dimensional arrays). The
general form for nested reductions on two-dimensional arrays is given as follows.
([f, ⊕, ⊗]) ◦ map ([g, ⊖, ⊙]) ◦ gg
Here, gg is a function to generate nested arrays. The generated arrays are consumed
by the nested reductions: ([f, ⊕, ⊗]) for the outer structure, and ([g, ⊖, ⊙]) for the
inner structures.
Note that homomorphism ([f, ⊕, ⊗]) is equivalent to the composition of skeletons:
reduce (⊕, ⊗) ◦ map f . We do not deal with filtering for two-dimensional arrays.
Here is an extension of the specifications. We will use a product of k homomorphisms, i.e., ([g1 , ⊖1 ]) × · · · × ([gk , ⊖k ]) and ([g1 , ⊖1 , ⊙1 ]) × · · · × ([gk , ⊖k , ⊙k ]) insteadof the simple homomorphism ([g, ⊖]) and ([g, ⊖, ⊙]), when generation functions generate nested structures in which generated inner structures are tupled.

5.3.2

Functions to Generate Nested Data Structures

This section gives a collection of useful functions to generate nested structures fromthe given flat structure, which will be used as the first function gg of the generalform. For each function, we will give the formal definition and example applications
written with the function.
Now, we will introduce some predicates used in the examples applications.
Predicate ascending returns True if the given list is ascending (i.e., each element issmaller than the following element). For example, applying ascending to a sorted list[−2, 4, 5] we get ascending [−2, 4, 5] = True. Conversely, ascending [3, 6, 2] = False
since 6 is not smaller than 2.
Predicate descending returns True if the given list is descending (i.e., each element is bigger than the following element). For example, applying descending toa descendingly sorted list [5, 3, 0] we get descending [5, 3, 0] = True. Conversely,
ascending [1, 9, 7] = False since 1 is not bigger than 9.
Predicate smooth c returns True if a difference between any successive elementsin the given list is less than or equal to c. For example, smooth 2 [6, 5, 7] = True,
since both of the differences 1 = |6 − 5| and 2 = |5 − 7| are less than or equal to
2. Conversely, smooth 2 [6, 5, 8] = False because the difference |5 − 8| = 3 is greaterthan 2.
Predicate high returns True if the maximum element of the given list is greaterthan the length of the list. For example, high [1, 4, 3] = True, since the maximum
element 4 is greater than the length of the list, i.e., 3. Conversely, high [1, 4, 3, 2] =False because the maximum element 4 is not greater than the length of the list 4.
Formal definitions of the above predicates will be shown later.

98

5. Domain-Specific Optimization for Skeleton Programs

Function inits to Generate Prefix Segments
Function inits generates all prefixes of an input list in the lexicographic order. For
example, applying inits to [1, 3, 1, −7, 2, 4], we get the following nested list.
inits [1, 3, 1, −7, 2, 4] = [[1], [1, 3], [1, 3, 1], [1, 3, 1, −7], [1, 3, 1, −7, 2], [1, 3, 1, −7, 2, 4]]Here, each element of the resulting list is a prefix of the given list.
The formal definition of inits is given as follows.
Definition 5.4 (Inits). Function inits is defined with homomorphism as follows.
inits = ([[·] ◦ [·], ⊕]) where x ⊕ y = x ++ map ((last x)++) y

The operator ⊕ in the above definition makes a list of initial segments of a list
u ++ v from lists (x and y in the above equation) of initial segments of u and v. Since
each initial segment of u is also an initial segment of u ++ v, x remains in the result.
Since each initial segment of v need to be concatenated with u to become an initial
segment of u ++ v, the operator maps ((last x)++) (u is the last element of x) to y.
The most famous application of inits is prefix sums, which has many applications [Ble90]. .
Example 5.5 (Prefix sums). The statement is as follows: Given a list and an
associative binary operator, find sums for all prefixes of the given list.
For example, prefix sums of [1, 3, 1, −7, 2, 4] with operator ⊕ is [1, 1 ⊕ 3, 1 ⊕ 3 ⊕1, 1 ⊕ 3 ⊕ 1 ⊕ −7, 1 ⊕ 3 ⊕ 1 ⊕ −7 ⊕ 2, 1 ⊕ 3 ⊕ 1 ⊕ −7 ⊕ 2 ⊕ 4]. Using inits, prefixsums are easily obtained by applying reduction with ⊕ to all prefixes generated byinits.
map (reduce (⊕)) (inits [1, 3, 1, −7, 2, 4])= map (reduce (⊕)) [[1], [1, 3], [1, 3, 1], [1, 3, 1, −7], [1, 3, 1, −7, 2], [1, 3, 1, −7, 2, 4]]= [1, 1 ⊕ 3, 1 ⊕ 3 ⊕ 1, 1 ⊕ 3 ⊕ 1 ⊕ −7, 1 ⊕ 3 ⊕ 1 ⊕ −7 ⊕ 2, 1 ⊕ 3 ⊕ 1 ⊕ −7 ⊕ 2 ⊕ 4]
Note that the outer reduction may be the identity function id = ([[·], ++]).
The maximum prefix (initial-segment) sum problem is one of optimization problems on sequences.
Example 5.6 (Maximum prefix sum). Its statement is as follows: Given a list, findthe maximum sum of a prefix of the given list.
For example, the maximum prefix sum of [1, 3, 1, −7, 2, 4] is 5 since its prefix sums
(with the usual plus operator +) are [1, 4, 5, −2, 0, 4]. So, using inits we can get the5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

99

maximum prefix sum easily by applying reduction with the maximum operator ↑for the prefix sums.
reduce (↑) (map (reduce (+)) (inits [1, 3, 1, −7, 2, 4]))
= reduce (↑) (map (reduce (+)) [[1], [1, 3], [1, 3, 1], [1, 3, 1, −7], [1, 3, 1, −7, 2],[1, 3, 1, −7, 2, 4]])
= reduce (↑) ([1, 1 + 3, 1 + 3 + 1, 1 + 3 + 1 + −7, 1 + 3 + 1 + −7 + 2,
1 + 3 + 1 + −7 + 2 + 4])=5
Here is a variant of the maximum prefix sum problem.
Example 5.7 (Maximum p-prefix sum). Its statement is as follows: Given a list
and a predicate p, find the maximum sum of its prefix satisfying the given predicatep.
For example, the maximum ascending-prefix sum of [1, 3, 1, −7, 2, 4] is 4 since its
ascending prefixes are [1] and [1, 3]. Using inits and filter , we can get the maximum
ascending-prefix sum easily as follows.
reduce (↑) (map (reduce (+)) (filter ascending (inits [1, 3, 1, −7, 2, 4])))= reduce (↑) (map (reduce (+)) (filter ascending [[1], [1, 3], [1, 3, 1], [1, 3, 1, −7],
[1, 3, 1, −7, 2], [1, 3, 1, −7, 2, 4]]))= reduce (↑) (map (reduce (+)) [[1], [1, 3]])= reduce (↑) [1, 1 + 3]
=4
All three examples above are instances of the general form. Here is the summaryof the example programs.
prefix sums
: ([[·], ++]) ◦ map (reduce (⊕)) ◦inits
maximum prefix sum: reduce (↑) ◦ map (reduce (+)) ◦inits
maximum p-prefix sum : reduce (↑) ◦ map (reduce (+)) ◦ filter p ◦inits
Function tails to Generate Suffix Segments
Function tails generates all suffixes of an input list in the lexicographic order1 . For
example, applying tails to [1, 3, 1, −7, 2, 4], we get the following nested list.
tails [1, 3, 1, −7, 2, 4] = [[1, 3, 1, −7, 2, 4], [3, 1, −7, 2, 4], [1, −7, 2, 4], [−7, 2, 4], [2, 4], [4]]Here, each element of the resulting list is a suffix of the given list.
The formal definition of tails is given as follows.
1

Note that the lexicographic order is that with respect to positions of elements in the input. The
following example make this clear: tails [a1 , a2 , a3 ] = [[a1 , a2 , a3 ], [a2 , a3 ], [a3 ]]. Here, concatenationsof the indices of the suffixes are 123, 23, and 3, which is arranged in the lexicographic order.

100

5. Domain-Specific Optimization for Skeleton Programs

Definition 5.8 (Tails). Function tails is defined with homomorphism as follows.
tails = ([[·] ◦ [·], ⊕]) where x ⊕ y = map (++(head y)) x ++ y

The operator ⊕ in the above definition makes a list of tail segments of a list
u ++ v from lists (x and y in the above equation) of tail segments of u and v. Since
each tail segment of v is also a tail segment of u ++ v, y remains in the result. Since
each tail segment of u need to be concatenated with v to become a tail segment ofu ++ v, the operator maps (++(head y)) (v is the head element of y) to x.
The most famous application of tails is suffix sums, which is converse computationof prefix sums shown in the previous section.
Example 5.9 (Suffix sums). The statement is as follows: Given a list and an
associative binary operator, find sums for all suffixes of the given list.
For example, suffix sums of [1, 3, 1, −7, 2, 4] with operator ⊕ is [1 ⊕ 3 ⊕ 1 ⊕ −7 ⊕2 ⊕ 4, 3 ⊕ 1 ⊕ −7 ⊕ 2 ⊕ 4, 1 ⊕ −7 ⊕ 2 ⊕ 4, −7 ⊕ 2 ⊕ 4, 2 ⊕ 4, 4] . Using tails, we canget suffix sums easily by applying reduction with ⊕ to all suffixes generated by tails.
map (reduce (⊕)) (tails [1, 3, 1, −7, 2, 4])= map (reduce (⊕)) [[1, 3, 1, −7, 2, 4], [3, 1, −7, 2, 4], [1, −7, 2, 4], [−7, 2, 4], [2, 4], [4]]= [1 ⊕ 3 ⊕ 1 ⊕−7 ⊕ 2 ⊕ 4, 3 ⊕ 1 ⊕−7 ⊕ 2 ⊕ 4, 1 ⊕−7 ⊕ 2 ⊕ 4,−7 ⊕ 2 ⊕ 4, 2 ⊕ 4, 4]
The maximum suffix sum problem is one of optimization problems on sequences.
Example 5.10 (Maximum suffix sum). Its statement is as follows: Given a list,
find the maximum sum of a suffix of the given list.
For example, the maximum suffix sum of [1, 3, 1, −7, 2, 4] is 6 since its suffix sums
(with the usual plus operator +) are [4, 3, 0, −1, 6, 4]. So, using tails we can get the
maximum suffix sum easily by applying reduction with the maximum operator ↑ forsuffix sums.
reduce (↑) (map (reduce (+)) (tails [1, 3, 1, −7, 2, 4]))
= reduce (↑) (map (reduce (+)) [[1, 3, 1, −7, 2, 4], [3, 1, −7, 2, 4], [1, −7, 2, 4],[−7, 2, 4], [2, 4], [4]]= reduce (↑) [1 + 3 + 1 + −7 + 2 + 4, 3 + 1 + −7 + 2 + 4, 1 + −7 + 2 + 4,
−7 + 2 + 4, 2 + 4, 4]=6Here is a variant of the maximum suffix sum problem.
Example 5.11 (Maximum p-suffix sum). Its statement is as follows: Given a list
and a predicate p, find the maximum sum of its suffix satisfying the given predicatep.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

101

For example, the maximum ascending-suffix sum of [1, 3, 1, −7, 2, 4] is 6 since its
ascending suffixes are [−7, 2, 4], [2, 4] and [4]. Using tails and filter , we can get the
maximum ascending-suffix sum easily as follows.
reduce (↑) (map (reduce (+)) (filter ascending (tails [1, 3, 1, −7, 2, 4])))= reduce (↑) (map (reduce (+)) (filter ascending [[1, 3, 1, −7, 2, 4], [3, 1, −7, 2, 4],[1, −7, 2, 4], [−7, 2, 4], [2, 4], [4]]))= reduce (↑) (map (reduce (+)) [[−7, 2, 4][2, 4], [4]])= reduce (↑) [−7 + 2 + 4, 2 + 4, 4]=6
Function segs to Generate All Segments
Function segs generates segments (continuous subsequences) of an input list in thelexicographic order. For example, applying segs to [3, 2, −7, 4, 2] we get the following
nested list.
segs [3, 2, −7, 4, 2] = [[3], [3, 2], [3, 2, −7], [3, 2, −7, 4], [3, 2, −7, 4, 2], [2], [2, −7],
[2, −7, 4], [2, −7, 4, 2], [−7], [−7, 4], [−7, 4, 2], [4], [4, 2], [2]]The formal definition of segs is given as follows.
Definition 5.12 (Segs). Function segs is defined with homomorphism as follows.
segs = reduce (++) ◦ map inits ◦ tails
This definition is based on the fact that a segment of a list is a prefix of a suffixof the list.
The most famous application of segs is maximum segment sum problem [Bir87,
Jeu93, SHTO00], which is one of optimization problems on sequences.
Example 5.13 (Maximum segment sum). The statement is as follows: Given a list,
find the maximum sum of a segment of the given list.
For example, the maximum segment sum of [3, 2, −7, 4, 2] is 6 that is the sum of
its segment [4, 2]. We can get the maximum segment sum of the given list using segs
as follows. First, we generate all segments of the given list by segs. Then, we apply
reduction with the usual plus operator + to generated segments to get all segment
sums. Finally, applying reduction with maximum-operator to those segment sums,
we get the maximum segment sum of the given list.
reduce (↑) (map (reduce (+)) (segs [3, 2, −7, 4, 2]))
= reduce (↑) (map (reduce (+)) [[3], [3, 2], [3, 2, −7], [3, 2, −7, 4], [3, 2, −7, 4, 2],[2], [2, −7], [2, −7, 4], [2, −7, 4, 2], [−7], [−7, 4],[−7, 4, 2], [4], [4, 2], [2]])= reduce (↑) [3, 5, −2, 2, 4, 2, −5, −1, 1, −7, −3, −1, 4, 6, 2]
=6

102

5. Domain-Specific Optimization for Skeleton Programs
Here is a variant of the maximum segment sum problem.

Example 5.14 (Maximum p-segment sum). Its statement is as follows: Given alist and a predicate p, find the maximum sum of its segment satisfying the given
predicate p.
For example, the maximum ascending-segment sum of [3, 2, −7, 4, 2] is 4 since
its ascending-segments are all singletons and [−7, 4]. The maximum descendingsegment sum of [3, 2, −7, 4, 2] is 6 since its descending-segments are [3, 2], [3, 2, −7],
[2, −7], [4, 2] and all singletons. Using segs and filter , we can get the maximum
descending-segment sum easily as follows.
reduce (↑) (map (reduce (+)) (filter descending (segs [3, 2, −7, 4, 2])))= reduce (↑) (map (reduce (+)) (filter descending [[3], [3, 2], [3, 2, −7], [3, 2, −7, 4], [3, 2, −7, 4, 2],[2], [2, −7], [2, −7, 4], [2, −7, 4, 2], [−7], [−7, 4],[−7, 4, 2], [4], [4, 2], [2]]))= reduce (↑) (map (reduce (+)) [[3], [3, 2], [3, 2, −7], [2], [2, −7], [−7], [4], [4, 2], [2]])= reduce (↑) [3, 5, −2, 2, −5, −7, 4, 6, 2]=6Longest-p segment problem [Zan92] is also one of optimization problems on sequences.
Example 5.15 (Longest-p segment). Its statement is as follows: Given a list and a
predicate, find the longest segment (continuous subsequence) of the list that satisfies
the predicate.
For example, the longest-ascending segment of [3, 2, −7, 4, 2] is [−7, 4], while
the longest-descending segment is [3, 2, −7]. There are many instances of longest-p
segment for various predicates. The longest segment satisfying a given predicate p
is obtained by using segs and filtering. Some instances are shown below.
The longest smooth segment of an input list is obtained by using the predicatesmooth c .
reduce (↑length ) (filter smooth 4 (segs [3, 2, −7, 4, 2]))
= reduce (↑length ) (filter smooth 4 (segs [[3], [3, 2], [3, 2, −7], [3, 2, −7, 4], [3, 2, −7, 4, 2],[2], [2, −7], [2, −7, 4], [2, −7, 4, 2], [−7], [−7, 4],[−7, 4, 2], [4], [4, 2], [2]]))= reduce (↑length ) [[3], [3, 2], [2], [−7], [4], [4, 2], [2]]= [3, 2]

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

103

The longest high segment of an input list is obtained by using predicate high.
reduce (↑length ) (filter high (segs [3, 2, −7, 4, 2]))
= reduce (↑length ) (filter high (segs [[3], [3, 2], [3, 2, −7], [3, 2, −7, 4], [3, 2, −7, 4, 2],[2], [2, −7], [2, −7, 4], [2, −7, 4, 2],[−7], [−7, 4], [−7, 4, 2], [4], [4, 2], [2]]))= reduce (↑length ) [[3], [3, 2], [2], [4], [4, 2], [2]]= [3, 2]Function neighbors to Generate Neighbor Segments
Function neighbors generates left and right neighbor segments for each element of an
input list. For example, applying neighbors to [1, 3, 1, −7, 2, 4], we get the following
list of tuples, in which each tuple consists of the left neighbor segment, the center
element, and the right neighbor segment.
neighbors [1, 3, 1, −7, 2, 4] = [([ ], 1, [3, 1, −7, 2, 4]), ([1], 3, [1, −7, 2, 4]),([1, 3], 1, [−7, 2, 4]), ([1, 3, 1], −7, [2, 4]),([1, 3, 1, −7], 2, [4]), ([1, 3, 1, −7, 2], 4, [ ])]The formal definition of neighbors is given as follows.
Definition 5.16 (Neighbors). Function neighbors is defined using inits and tails as
follows.
neighbors = zipP 3 ◦ ((map init ◦ inits)△id △(map tail ◦ tails))
where zipP 3 (x, y, z) = zip x y z

Basically, neighbors is a composition of inits and tails. It uses extra init and tailto remove redundant occurrence of the center element.
Here is a variant of neighbors to generate neighbor segments of fixed sizes. Thisvariant is useful for describing examples shown in the previous section (Section 5.2).
For example, applying the variant neighbors ′ to [1, 3, 1, −7, 2, 4] with boundaries[5, 6] and [9], we get the following list of tuples. Here, each left neighbor segment
has two elements, and each right neighbor segment has only one element, which are
the same as the given boundaries.
neighbors ′ [5, 6] [9] [1, 3, 1, −7, 2, 4] = [([5, 6], 1, [3]), ([6, 1], 3, [1]), ([1, 3], 1, [−7]),([3, 1], −7, [2]), ([1, −7], 2, [4]), ([−7, 2], 4, [8])]The formal definition of neighbors ′ is given as follows.

104

5. Domain-Specific Optimization for Skeleton Programs

Definition 5.17 (Finite-window neighbors). Function neighbors ′ takes two lists forboundary elements: ls for the left edge, and rs for the right edge.
neighbors ′ ls rs = map (taker m ◦ (ls++) × id × take n ◦ (++rs)) ◦ neighbors
where m = length ls
n = length rs

Example 5.18 (Running example in Section 5.2). The running example in Section 5.2 is described with neighbors as follows.
next = map f ◦ neighbors ′ [bl0 , bl1 ] [br ]where f ([u−2 , u−1 ], u0 , [u1 ]) = c−2 × u−2 + c−1 × u−1 + c0 × u0 + c1 × u1
Function subs to Generate All Subsequences
Function subs generates all subsequences (sub-lists) of an input list. Generated
subsequences are listed in the lexicographic order. Its definition is as follows.
subs [a]= [[a]]
subs (a : x) = let sx = subs x in [a] ++ map ([a]++) sx ++ sx
For example, applying subs to [2, 5, −3, 4] we get the following nested list.
subs [2, 3, −3, 4] = [[2], [2, 3], [2, 3, −3], [2, 3, −3, 4], [2, 3, 4], [2, −3], [2, −3, 4], [2, 4],[3], [3, −3], [3, −3, 4], [3, 4], [−3], [−3, 4], [4]]Longest-p subsequence problem is one of optimization problems on sequences [Jeu93].
Example 5.19 (Longest-p subsequence). Its statement is as follows: Given a list
and a predicate, find the longest subsequence of the list that satisfies the predicate.
For example, the longest-ascending subsequence of [2, 3, −3, 4] is [2, 3, 4], while
the longest-descending subsequence is [2, −3]. There are many instances of longestp subsequence for various predicates. The longest subsequence satisfying a given
predicate p is obtained by using subs and filter . Some instances are shown blow.
The longest ascending subsequence of an input list is obtained by using subs andthe predicate ascending.
reduce (↑length ) (filter ascending (subs [2, 3, −3, 4]))
reduce (↑length ) (filter ascending[[2], [2, 3], [2, 3, −3], [2, 3, −3, 4], [2, 3, 4], [2, −3], [2, −3, 4], [2, 4],[3], [3, −3], [3, −3, 4], [3, 4], [−3], [−3, 4], [4]])= reduce (↑length ) [[2], [2, 3], [2, 3, 4], [2, 4], [3], [3, 4], [−3], [−3, 4], [4]]= [2, 3, 4]

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

105

The longest descending subsequence of an input list is obtained by using subs andthe predicate descending.
reduce (↑length ) (filter descending (subs [2, 3, −3, 4]))
reduce (↑length ) (filter descending[[2], [2, 3], [2, 3, −3], [2, 3, −3, 4], [2, 3, 4], [2, −3], [2, −3, 4],[2, 4], [3], [3, −3], [3, −3, 4], [3, 4], [−3], [−3, 4], [4]])= reduce (↑length ) [[2], [2, −3], [3], [3, −3], [−3], [4]]= [2, −3]
Here is another example problem.
Example 5.20 (0-1 knapsack problem). Given a list of items (pairs of value andweight) and a knapsack of fixed capacity, find a subset of items that has the maximum sum of values and its sum of weight is less than or equals to the capacity.
For example, given four items [(2, 1), (5, 3), (1, 1), (4, 2)] (first element of a pair isvalue and the second is weight) and a knapsack of capacity 4, the solution (a set ofitems to be put into the knapsack) is [(2, 1), (5, 3)]. This solution is given as follows.
First, we generate every combination of items by subs. Then, using filtering, wethrow away combinations of which weight is greater than the capacity. Finally, wetake the combination that have the maximum sum of values.
reduce (↑reduce (+) ◦ map π1 ) (filter ((≤ 4)◦(reduce (+))◦(map π2 )) (subs [(2, 1), (5, 3), (1, 1), (4, 2)]))= reduce (↑reduce (+) ◦ map π1 ) [[(2, 1)], [(2, 1), (5, 3)], [(2, 1), (1, 1)], [(2, 1), (1, 1), (4, 2)],[(2, 1), (4, 2)], [(5, 3)], [(5, 3), (1, 1)], [(1, 1)], [(1, 1), (4, 2)], [(4, 2)]]= [(2, 1), (5, 3)]

Functions Ts, Ls, Bs, Rs, and Their Combinations to Generate Rectangles
We will introduce eight functions to generate various rectangles. Basically, thesefunctions are extensions of the functions inits and tails on lists.

−
◦

−
◦

−
◦

scan (−◦, ≫) ◦ map | · |scan (≫, ) ◦ map | · |scanr (−◦, ≪) ◦ map | · |scanr (≪, ) ◦ map | · |scan (−◦, ) ◦ map | · |◦, ≫) ◦ map | · |scanr (≪, ) ◦ scan (−scanr (−◦, ≪) ◦ scan (≫, ) ◦ map | · |scanr (−◦, ) ◦ map | · |
−
◦

−
◦

=
=
=
=
=
=
=
=

−
◦

Ts
Ls
Bs
Rs
TLs
TRs
BLs
BRs

106

5. Domain-Specific Optimization for Skeleton Programs

Example use of those functions are shown below.


 


2
7
3
     
2 7 3
3 
7
=  2
Ts
8 4 1
1
4
8



 

2  2 7 2 7 3
2 7 3
=
Ls
8
8 4
8 4 1
8 4 1
     


3
7
2
2 7 3

1 
4
8
=
Bs
8 4 1
8
4
1






2 7 3 7 3  3
2 7 3
=
Rs
8 4 1
4 1
1
8 4 1


 


2
2
7
2
7
3

 
  
2 7 3
2 7 3 
2 7
=  2
TLs
8 4 1
8 4 1
8 4
8






2 7 3 7 3 3

2 7 3
3 
7 3
=  2 7 3
TRs
8 4 1
1
4 1
8 4 1

 
  


2 7 3
2 7
2
2 7 3
8 4 1 8 4
=  8
BLs
8 4 1
8
8 4
8 4 1

  
 
3
7 3
2 7 3


2 7 3
 8 4 1
1 
4 1
= 
BRs



8 4 1
8 4 1
4 1
1

For each element of the input array, the first four functions generate rectangles
consisting of elements on its top, left, bottom, and right, respectively. The rest forfunctions generate rectangles consisting of elements on its top left, top right, bottom
left, and bottom right, respectively.
Those functions can be used to perform directed computations on arrays.
Example 5.21 (Top-left prefix sums). Given a two-dimensional array, two associative, abiding binary operators, find sums forall top-left
 prefixes of the given array.
2 7 3with + operators is given as
For example, the top-left prefix sums of
8 4 1
follows.



2 7 3
map (reduce (+, +)) TLs
8 4 1




reduce (+, +) 2 reduce (+, +) 2 7 reduce (+, +) 2 7 3
2 7 3 2 7
2
= 
reduce (+, +)reduce (+, +)reduce (+, +)
8 4 1
8 4
8


2 9 12=
10 21 25

107

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements
Function surrounds to Generate Surrounding Rectangles

For each element in the given array, the function surrounds generates the all rectangles surrounding the element. This function corresponds to the simultaneous use of
the previously-defined eight functions Ts, Ls, Bs, Rs, TLs, TRs, BLs, and BRs, exceptthat each rectangle generated by surrounds does not contain the element itself.
The function surrounds is defined by the following two-phase computation: (1)computation of the parts of the northwest (i.e. c, n, w and nw) by scan, and (2)that of the other parts by scanr.
surrounds = scanr(⊕r , ⊗r ) ◦ map fr ◦ scan(⊕f , ⊗f ) ◦ map ff
where
ff a = (a, NIL, NIL, NIL)◦|c |−◦n , wb , nwa−
(ca , na , wa , nwa ) ⊕f (cb , nb , wb , nwb ) = ( cb , na−◦wa−◦nwb )|{z} | {za }b |{z}
{z
}
|
c

nw

w

n

c

n

−
◦

−
◦

−
◦

−
◦

(ca , na , wa , nwa ) ⊗f (cb , nb , wb , nwb ) = ( cb , nb , wa |ca | wb , nwa na nwb )|{z} |{z} |
{z
}
{z
} |
nw

w

fr (c, n, w, nw) = (c, n, NIL, NIL, w, NIL, nw, NIL, NIL)
(ca , na , sa , ea , wa , nea , nwa , sea , swa ) ⊕r (cb , nb , sb , eb , wb , neb , nwb , seb , swb )◦ eb −◦ seb , swa −◦ |cb |−◦ sb , ea , wa , nea , nwa , sea −◦ wb −◦ swb )= ( c a , na , s a −
{z
} |
{z
}
|{z} |{z} |{z} |{z} |{z} |{z} |{z} |
c

n

e

s

w

ne

nw

se

sw

c

s

n

e

w

ne

nw

se

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

(ca , na , sa , ea , wa , nea , nwa , sea , swa ) ⊗r (cb , nb , sb , eb , wb , neb , nwb , seb , swb )= ( ca , na , sa , ea |cb | eb , wa , nea nb neb , nwa , sea sb seb , swa ){z} |{z} |{z} |{z}|{z} |{z} |{z} |{z} |{z} |
sw

−
◦

−
◦

−
◦

Here, NIL is a special value to indicate that there is no value, and we treat it as an
identity of−◦ and for simplification of the notation. Thus, NIL−
◦ x = x, x−◦ NIL = x,
NIL x = x, and x NIL = x. Each element of the resulting array is a tuple of nineelements. The meaning of each element of the tuple is as follows: c is the center
element; s is an array of the elements on the south of the element; similarly n, e
and w are arrays of the elements on the north, east and west respectively; ne, nw,
se and sw are arrays of the elements on the northeast, northwest, southeast andsouthwest.
The function surrounds can be used to describe computations known as matrixconvolutions [Jai89, Rus06, GW06], in which each element in the resulting arraydepends on its surrounding elements. This set of computations includes importantand fundamental problems such as image filters, difference methods, and the N -bodyproblem.

Example 5.22 (Image filter). An image filter by matrix-convolution is described

108

5. Domain-Specific Optimization for Skeleton Programs

2
0
6
w 6 3
66
66 4
sw 4 6

nw

6

2
0
6
3
6
6
6
4
6
6
46
6

2

2

6

4

3

3

5

5

5

5

5

9

3
2
6
Æ
7
6
6 7 surrounds 6 Æ
7
66
6 7
77
66 Æ
7 5
4Æ

6

6

7

7

n
2

2

6

4

3

3

5

5

5

5

5

9

3 ne
7
6 7 e
7
6 7
shrink77
7 5 se

6

6

7

7

s

6

2
0
6
3
6
6
6
4
6
6
46
6

3

2

2

2

6

4

3

3

5

5

5

5

5

9

3
7
6 7
7
6 7
7
7
7 5

6

6

7

7

6

3

2
0
sum Æ zipwith() 64 1

Æ Æ Æ ÆÆ Æ Æ Æ ÆÆ Æ Æ Æ 777 map shrink 666 Æ Æ Æ Æ Æ 777 map f6Æ Æ Æ Æ 77
Æ Æ Æ Æ Æ 77676
Æ Æ Æ Æ5
5
4Æ Æ Æ Æ Æ7
Æ Æ Æ Æ ÆÆ Æ Æ Æ Æ

0

2
5 4
6
7
7
6
6
6
6
7
6
6
4 15 3
18

13

1
5
1

3

0
7
1 5
0

19

18

0

3
7
5 15 77
2 12 77
7
21 13 5

12

13

21

1
1
7

Figure 5.6. An image of the sharpen filter written with surroundswith surrounds as follows.

−
◦

−
◦

imagefilter ker = map (conv ker ) ◦ map shrink 1 ◦ surrounds
whereshrink 1 = id × B × T × L × R × BL × BR × TL × TR
B = ([| · |, ≫, ]), T = ([| · |, ≪, ]),L = ([| · |,−◦, ≪ ]), R = ([| · |,−◦, ≫ ]),BL = ([| · |, ≫, ≪ ]), BR = ([| · |, ≫, ≫ ]), TL = ([| · |, ≪, ≪ ]), TR = ([| · |, ≪, ≫ ])The function imagefilter ker is an image filter with the coefficient matrix ker , which
is used to compute weighted sum of the surrounding pixels. The shrink 1 reduceseach part of the gathered surrounding elements to the element closest to the center,and the function conv ker calculates the weighted sum of them. The functions B andT take the bottom row and the top row of the input array respectively. Similarly,
each of L, R, BL, BR, TL and TR takes corresponding part of the input array.
Figure 5.6 shows an image of execution of the sharpen-filter by the above generalprogram.
The following example computes for each element the maximum of the column
and the row in which the element belongs to.
Example 5.23 (Cross maximum). Given a two-dimensional arrays, find for eachelement the maximum in the cross of the column and the row that the element
belongs to.
The cross max can be computed by the following program. The shrink max reduceseach part of the gathered surrounding elements to the biggest element in the part.
The function max 5 takes the maximum of the column and the row including the5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

109

center element.
crossmax = map max 5 ◦ map shrink max ◦ surrounds
where shrink max = max × · · · × max
max = ([id , ↑, ↑ ])max 5 (c, n, s, e, w, , , , ) = c ↑ n ↑ s ↑ e ↑ w
Function rects ′ to Generate All Rectangles
Finally, we will introduce the function rects ′ to generate all rectangles in the given
array.
rects ′ = flatten ◦ map TLs ◦ BRs


2 7 3
′
, we get the following result.
For example, applying rects to
8 4 1


2 7 3
′
rects
8 4 1

  
 
3
7 3
2 7 3
 8 4 11 
4 1
= (flatten ◦ map (TLs)) 



8 4 1
4 1
1
      
 
3
7  7 3 
2  2 7  2 7 3 
 2




3 
7 3
7
2 7 3
2 7

= flatten 

1 
4 1
4
8 4 
18  8 4
8
8 4
8 4 1
4
4 1
1





 
2 2 7 2 7 3 7 7 3 3
 2
3 
7 3
7
2 7 3
2 7

= 

 8
1
4
1
4
8
4
1
8
4






8
8 4
8 4 1
4
4 1
1

The example application described with rects ′ is the maximum rectangle problem, which has already been discussed in Section 4.3.4. This is a simplified problemof pattern matching in two-dimensional data structures.
Example 5.24 (Maximum rectangle sum). Its statement is as follows: Given a
two-dimensional array, find the maximum of sums of all the rectangle areas in thearray. This problem was originated by Bentley [Ben84a, Ben84b] and improved by
Takaoka [Tak02].
For example, the answer is 15 for the following data.

3 −1
4 −1 −5 1 −4 −1
5 −3 
−41
5
31

Here, the sub-rectangle contributing the answer is denoted by bold numbers. The
program to solve the problem is given as follows.
reduce (↑, ↑) ◦ map (reduce (+, +)) ◦ rects ′

110

5.3.3

5. Domain-Specific Optimization for Skeleton Programs

Useful Properties on Predicates

We will define some properties on predicates, which will be used in development of
optimization theorems for each function shown in the previous section.
The following closure properties are defined on predicates for deriving efficientsequential implementation [Zan92, Jeu93]. Here, ⇒ means the implication.
Definition 5.25 (Prefix-closed predicate). Predicate p is said to be prefix-closed if
the following equation holds for all x and y.
p (x ++ y) ⇒ p (x)Definition 5.26 (Suffix-closed predicate). Predicate p is said to be suffix-closed if
the following equation holds for all x and y.
p (x ++ y) ⇒ p (y)Definition 5.27 (Segment-closed predicate). Predicate p is said to be segmentclosed if p is prefix- and suffix closed, i.e. the following equation holds for all x, y
and z.
p (x ++ y ++ z) ⇒ p (y)Basically, those properties guarantee that we can know the result of the predicate
for a long list from the result for smaller segments of the list. The following property
is the almost converse of the segment-closedness.
Definition 5.28 (Overlap-closed predicate). Predicate p is said to be overlap-closed
if the following equation holds for all x, y and z.
p (x ++ y) ∧ p (y ++ z) ∧ y 6= [ ] ⇒ p (x ++ y ++ z)
Prefix-closed property plays the most important role in derivation of efficientsequential implementation of segment problems [Zan92, Jeu93]. Other properties
are used as auxiliary tools to improve the derived implementation. However, inderivation of efficient parallel implementation, the pair of segment-closed property
and overlap-closed property plays the most important role, which will be shown
later.
A candidate of segment-closed and overlap-closed predicates is given by a relation [Zan92].

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

111

Definition 5.29 (Relational predicate). Given a relation R, relational predicate pR
is defined as follows.
VpR (x) = {aRb | [a, b] ∈ segments x}

Here, segments x returns a set of segments (contiguous subsequences) of x, i.e.,
segments x = { y | u ++ y ++ v = x }. Note that [a, b] ∈ segments x means thata and b are successive elements in x, since segments x generates all contiguous
subsequences of x and [a, b] ∈ segments x takes such subsequences of two elements.
Relational predicate pR is true for the given list x, if all successive elements a and b
in x satisfy the relation R, i.e. aRb = True.
We can also show the converse, i.e., a segment-closed, overlap-closed predicate isa relational predicate. The following lemma shows the relation between relational
predicates and segment-closed, overlap-closed predicates.
Lemma 5.30 (Relational predicate). Given predicate p that is true for all singletonsand empty list, the following statements are equivalent.
1. p is segment-closed and overlap-closed.
2. p is relational.
Proof. 2 ⇒ 1) Since p is relational, there exists a relation R and the following equation
holds.

p (x) =^

{aRb | [a, b] ∈ segments x}

First, we show that p is segment-closed.
p (x ++ y ++ z)= V{ unfolding p }{aRb | [a, b] ∈ segments (x ++ y ++ z)}⇒ V{ segments y ⊆ segments (x ++ y ++ z) }{aRb | [a, b] ∈ segments y}= { folding p }p (y)Next, we show that p is overlap-closed.
p (x ++ y) ∧ p (y ++ z) ∧ y 6= [ ]V{ unfolding p }V{aRb | [a, b] ∈ segments (x ++ y)} ∧ {aRb | [a, b] ∈ segments (y ++ z)} ∧ y 6=[ ] y 6= [ ] ⇒{[a, b] | [a, b] ∈ segments (x ++ y)} ∪ {[a, b] | [a, b] ∈ segments (y ++ z)}⇒= {[a, b] | [a, b] ∈ segments (x ++ y ++ z)}
V{aRb | [a, b] ∈ segments (x ++ y ++ z)}= { folding p }p (x ++ y ++ z)=

112

5. Domain-Specific Optimization for Skeleton Programs1 ⇒ 2) Letting R = {(a, b) | p ([a, b])}, we show p = pR by induction.
For base cases, we have p ([ ]) = p ([a]) = pR ([]) = pR ([a]) = True by assumption.
For induction case, we have p ([a] ++ x) = pR ([a] ++ x) by the following calculation.
p ([a] ++ x)= { segment-closed and overlap-closed }p (x) ∧ p ([a] ++ [head x])= { induction hypothesis and definition of R }pR (x) ∧ aR(head x)= { definition of pR }pR ([a] ++ x)

Thus, p = pR and p is relational.

In the above lemma, we assumed that the predicate p is true for all singletonsand empty list for simplicity. However, we can remove this assumption by letting
values of relational predicate pR in the proof be those of the given predicate p.
The following predicates are examples of relational predicates.
ascending (x)descending (x)flat (x)smooth c (x)

=
=
=
=

p< (x)
p> (x)
p= (x)pRc (x) where aRc b = |a − b| ≤ cPredicate ascending is true when the given list is ascendingly sorted, while descendingis true for descendingly sorted lists. Predicate flat is true if the all elements in thegiven list are the same. Predicate smooth c is true if the maximum of differences ofsuccessive elements is less than or equal to c. Especially, flat = smooth 0 .
It is worth mentioning about composition of predicates [Zan92]. Each closure
property of prefix-closed, suffix-closed, segment-closed and overlap-closed is closed
under disjunction. Each closure property of prefix-closed, suffix-closed, and segmentclosed is closed also under conjunction, but overlap-closed property is not closed
under conjunction. For example, ascending and descending are both overlap-closed,
but ascending ∨ descending is not overlap-closed since we can make a counterexample: ascending(x ++ y) ∧ descending(y ++ z) ∧ y 6= [ ] implies neither ascending(x ++y ++ z) nor descending(x ++ y ++ z).

5.3.4

Optimization Theorems for the Nested Reductions

In this section, we will give optimization theorems for nested reductions describedwith the generation functions discussed so far. We will develop a theory of optimizations for each generation function one by one, because in our theory the data
structure and the computation structure are closely related to each other, and thestructures of nested data are determined by the generation functions.
First, we will give theories for nested reductions on lists. Then, we will proceed
to theories for two-dimensional arrays.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

113

Optimization Theory of inits
In this section, we will give a theory for optimization of nested reductions describedwith inits.
The following two lemmas are well-known lemmas of inits [Bir87].
Lemma 5.31 (Inits-map promotion). For function f , the following equation holds.
map (map f ) ◦ inits = inits ◦ map f
This lemma gives us a way to promote the application of function f throughinits. The number of applications of function f on the left hand side is n(n + 1)/2,
while that on the right hand side is n. So, transformation from the left hand side
to the right hand side improves the efficiency. Also, the lemma enables us to ignore
a function g in the generate-and-test specification ([f, ⊕]) ◦ map ([g, ⊗]) ◦ filter p ◦inits,
since we can replace g with the identity function id and apply g to the input of inits
beforehand (of course, we need some tricks to move through filter p).
Lemma 5.32 (Scan). For any associative binary operator ⊕, the following equation
holds.
scan (⊕) = map (reduce (⊕)) ◦ inits
This lemma gives us the relation between computations of prefix sums by scanand by nested reductions with inits. It means that we can use scan to computenested reductions if the outer reduction is id = ([[·], ++]), which results in the smaller
number of uses of the operator ⊕.
The following theorem gives efficient parallel implementation of nested reductions
for inits when two reductions have distributivity. One of the most famous problems
for which this theorem is applicable is maximum initial-segment sum problem (alsoknown as maximum prefix sum problem), which is an instance of maximum marking
problems [SHTO00, Bir01].
Theorem 5.33 (Maximum initial-segment sum). Provided that ⊕ is associative,and ⊗ is associative and left-distributive over ⊕, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ inits = π1 ◦ ([pair , ⊙])where (i1 , s1 ) ⊙ (i2 , s2 ) = (i1 ⊕ (s1 ⊗ i2 ), s1 ⊗ s2 )pair a = (a, a)
Proof. We show the theorem by induction.

114

5. Domain-Specific Optimization for Skeleton Programs
For base case, we have LHS [a] = RHS [a] by the following calculation.

LHS [a]{ LHS }
(reduce (⊕) ◦ map (reduce (⊗)) ◦ inits) [a]= { definition of inits, map , and reduce }a= { definition of π1 , homomorphism, and pair }(π1 ◦ ([pair , ⊙])) [a]= { RHS }RHS [a]=For induction case, we have LHS (x ++ y) = RHS (x ++ y) by the following
calculation.

=
=
=

=

=
=
=
=
=

LHS (x ++ y){ LHS }
(reduce (⊕) ◦ map (reduce (⊗)) ◦ inits) (x ++ y){ definition of function composition, unfolding inits, last (inits x) = x }(reduce (⊕) ◦ map (reduce (⊗))) (inits x ++ map (x++) (inits y)){ definition of function composition, map and reduce }reduce (⊕) (map (reduce (⊗)) (inits x))⊕reduce (⊕) (map ((reduce (⊗)) x⊗)(map (reduce (⊗)) (inits y))){ distributivity of ⊗ }reduce (⊕) (map (reduce (⊗)) (inits x))⊕(reduce (⊗) x ⊗ reduce (⊕) (map (reduce (⊗)) (inits y))){ induction hypothesis }(π1 ◦ ([pair , ⊙])) x ⊕ (reduce (⊗) x ⊗ (π1 ◦ ([pair , ⊙])) y){ reduce (⊗) x = π2 (([pair , ⊙]) x) (shown below) }(π1 ◦ ([pair , ⊙])) x ⊕ ((π2 (([pair , ⊙]) x)) ⊗ (π1 ◦ ([pair , ⊙])) y){ definition of ⊙ }π1 (([pair , ⊙]) x ⊙ ([pair , ⊙]) y){ definition of homomorphism, and function composition }(π1 ◦ ([pair , ⊙])) (x ++ y){ RHS }RHS (x ++ y)5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

115

Finally, we show that reduce (⊗) x = π2 (([pair , ⊙]) x). For base case, we have
LHS [a]{ LHS }
reduce (⊗)[a]= { definition of reduce }a= { definition of π2 , homomorphism, and pair }π2 (([pair , ⊙]) [a])= { RHS }RHS [a]=For induction case, we have

=
=
=
=
=
=

LHS (x ++ y){ LHS }
reduce (⊗) (x ++ y){ definition of reduce }reduce (⊗) x ⊗ reduce (⊗) y{ induction hypothesis }π2 (([pair , ⊙]) x) ⊗ π2 (([pair , ⊙]) y){ definition of ⊙ }π2 (([pair , ⊙]) x ⊙ ([pair , ⊙]) y){ definition of homomorphism }π2 (([pair , ⊙]) (x ++ y)){ RHS }RHS (x ++ y)

The resulting program (the right hand side) of the theorem uses only one reduction with the new operator ⊙, while the original program (the left hand side) uses
two nested reductions with inits. The new operator ⊙ is applied to tuples. The firstelement of a tuple is equal to the result of the original program. The second elementof the tuple is equal to the reduction of the same input with operator ⊗, which isused to improve efficiency of the program by reusing the partial results effectively.
The resulting program (the right hand side of the equation) is more efficient thanthe original program. This is because the cost of the new operator is proportional
to the cost of operators in the original reductions. Also, the resulting program has
no intermediate data structures.
The following theorem extends Theorem 5.33, which allows filtering with thegiven predicate. It gives efficient parallel implementation of nested reductions forinits when two reductions have distributivity and the predicate is relational.

116

5. Domain-Specific Optimization for Skeleton Programs

Theorem 5.34 (Maximum p-initial-segment sum). Provided that ⊕ is associative,⊗ is associative and left-distributive over ⊕, and predicate p is relational, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits = π1 ◦ ([pentuple, ])
where(i1 , s1 , h1 , l1 , p1 )  (i2 , s2 , h2 , l2 , p2 ) =(i, s1 ⊗ s2 , h1 ≪ h2 , l1 ≫ l2 , p1 ∧ p2 ∧ p ([l1 , h2 ]))where i = i1 ⊕ if p1 ∧ p ([l1 , h2 ]) then s1 ⊗ i2 else ı⊕
pentuple a = (a, a, a, a, T )Proof. We show the theorem by induction.
For base case, we have

=
=
=
=

LHS [a]{ LHS }
(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits) [a]{ definition of inits, map , reduce and filter , p is true for singleton }a{ definition of π1 , homomorphism, and pentuple }(π1 ◦ ([pentuple, ])) [a]{ RHS }RHS [a]For induction case, we have the following calculation.
LHS (x ++ y){ LHS }
(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits) (x ++ y)= { definition of inits }(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p) (inits x ++ (map (x++) (inits y)))= { definition of filter , reduce, and map }(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits) x⊕ (reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p) (map (x++) (inits y))=

To proceed more, we calculate a part of the above equation filter p (map (x++) (inits y))
as follows.
filter p (map (x++) (inits y))= { p is relational: p (x ++ y) ⇒ p (y) }filter p (map (x++) (filter p (inits y)))= { p is relational: p (x ++ y) = p (y) ∧ p (x) ∧ p ([last x, head y]) }if p (x) ∧ p ([last x, head y]) then map (x++) (filter p (inits y)) else [ ]5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

117

Using this result, we proceed a wider part of the above equation as follows.
(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p) (map (x++) (inits y))= { above calculation }(reduce (⊕) ◦ map (reduce (⊗)))(if p (x) ∧ p ([last x, head y]) then map (x++) (filter p (inits y)) else [ ])= { distributivity of if-then-else, definition of map and reduce }if p (x) ∧ p ([last x, head y])then reduce (⊕) (map ((reduce (⊗)) x⊗) (map (reduce (⊗)) (filter p (inits y)))))
else ı⊕= { distributivity of ⊗ }if p (x) ∧ p ([last x, head y])then reduce (⊗) x ⊗ reduce (⊕) (map (reduce (⊗)) (filter p (inits y))) else ı⊕

Now, we resume the suspended calculation for induction case.

LHS (x ++ y)= { resume }(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits) x⊕ (reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p) (map (x++) (inits y))= { above calculation }(reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits) x⊗ if p (x) ∧ p ([last x, head y])then reduce (⊗) x ⊗ reduce (⊕) (map (reduce (⊗)) (filter p (inits y)))else ı⊕= { induction hypothesis }(π1 ◦ ([pentuple, ])) x⊗ if p (x) ∧ p ([last x, head y]) then reduce (⊗) x ⊗ (π1 ◦ ([pentuple, ])) y else ı⊕= { ( , reduce (⊗) x, head x, last x, p (x)) = ([pentuple, ]) x (shown below) }(π1 ◦ ([pentuple, ])) x⊗ if (π5 ◦ ([pentuple, ])) x ∧ p ([(π4 ◦ ([pentuple, ])) x, (π3 ◦ ([pentuple, ])) y])then (π2 ◦ ([pentuple, ])) x ⊗ (π1 ◦ ([pentuple, ])) y else ı⊕= { definition of  }(π1 ◦ ([pentuple, ])) (x ++ y)= { RHS }RHS (x ++ y)Finally, we show ( , reduce (⊗) x, head x, last x, p (x)) = ([pentuple, ]) x. For basecase, we have
( , reduce (⊗) [a], head [a], last [a], p ([a]))= { definition of each function, p is relational }( , a, a, a, T )= { definition of homomorphism and pentuple }([pentuple, ]) [a]118

5. Domain-Specific Optimization for Skeleton Programs

For induction case, we have
( , reduce (⊗) (x ++ y), head (x ++ y), last (x ++ y), p (x ++ y))= { definition of each function, p is relational }( , reduce (⊗) x ⊗ reduce (⊗) y,head x ≪ head y, last x ≫ head y, p x ∧ p y ∧ p ([head x, last y]))= { definition of  }( , reduce (⊗) x, head x, last x, p x)  ( , reduce (⊗) y, head y, last y, p y)
= { induction hypothesis }([pentuple, ]) x  ([pentuple, ]) y= { definition of homomorphism }([pentuple, ]) (x ++ y)Thus, we have ( , reduce (⊗) x, head x, last x, p (x)) = ([pentuple, ]) x.

Similar to the previous theorem, the resulting program (the right hand side) of
the theorem uses only one reduction with the new operator , while the original
program (the left hand side) uses two nested reductions with inits. The cost of the
new operator is proportional to the cost of operators in the original reductions andthe application of p. So, the resulting program is more efficient than the original
program.
The new operator  is applied to pentuples. The first element of a pentuple isequal to the result of the original program. The second element of the pentuple isequal to the reduction of the same input with operator ⊗, which is used to improve
efficiency of the program by reusing the partial results effectively. The third andforth elements are the edge elements of the input. Those edge elements are used tocheck whether results from two recursions in divide-and-conquer computation canbe connected to make a better solution. Since the predicate p is relational, we cancheck the connectability by using only elements on the edge. The fifth element is a
Boolean value that is the result of p applied to the input.
Here, we can reduce the size of pentuples by eliminating the fifth element (itcorresponds to p (x)) for simplicity, when we introduce an assumption that ı⊗ is thezero of ⊗, i.e. ı⊕ ⊗ a = ı⊕ .

Corollary 5.35 (Maximum p-initial-segment sum (simplified)). Provided that ⊕ is
associative, ⊗ is associative and left-distributive over ⊕, the identity ı⊕ is the zeroof ⊗, and predicate p is relational, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ inits = π1 ◦ ([quadruple, ⊠])where(i1 , s1 , h1 , l1 ) ⊠ (i2 , s2 , h2 , l2 ) = (i1 ⊕ (s1 ⊗ i2 )l1 ,h2 , (s1 ⊗ s2 )l1 ,h2 , h1 ≪ h2 , l1 ≫ l2 )
quadruple a = (a, a, a, a)(a)l,h = if p ([l, h]) then a else ı⊕5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

119

Proof. To simplify the result of the theorem, we add an invariant to the result ofTheorem 5.34. The invariant added to the pentuple (i, s, h, t, p) is ¬p ⇒ s = ı⊕ .
In the following calculation, we derive a operator slightly changed from that ofTheorem 5.34.
=
=
=
=
=

i
{ definition }i1 ⊕ if p1 ∧ p ([l1 , h2 ]) then s1 ⊗ i2 else ı⊕{ splitting condition }i1 ⊕ if p ([l1 , h2 ]) then (if p1 then (s1 ⊗ i2 ) else ı⊕ ) else ı⊕{ assumption: ı⊕ is the zero }i1 ⊕ if p ([l1 , h2 ]) then ((if p1 then s1 else ı⊕ ) ⊗ i2 ) else ı⊕{ s′1 = if p1 then s1 else ı⊕ }i1 ⊕ if p ([l1 , h2 ]) then (s′1 ⊗ i2 ) else ı⊕{ definition of (a)t,h }i1 ⊕ (s′1 ⊗ i2 )l1 ,h2Computation of s′ = if p then s else ı⊕ is as follows.
=
=
=
=
=

s′
{ definition }if p then s else ı⊕{ computation of s and p }if p1 ∧ p2 ∧ p ([l1 , h2 ]) then s1 ⊗ s2 else ı⊕{ splitting condition }if p ([l1 , h2 ]) then (if p1 then s1 else ı⊕ ) ⊗ (if p1 then s2 else ı⊕ ) else ı⊕{ definition of s’ }if p ([l1 , h2 ]) then s′1 ⊗ s′2 else ı⊕{ definition of (a)t,h }(s′1 ⊗ s′2 )l1 ,h2

Now, we can use s′ instead of s and p in the pentuple (i, s, h, t, p), since p and s are
not used by computation of other parts. Thus, replacing s and p in the pentuple
with s′ and rename s′ as s, we get the reduced operator ⊠.
The resulting program (the right hand side) uses the new reduction operator ⊠that is applied on quadruples. The difference from the result of Theorem 5.34 is thatthe second element becomes the identity ı⊕ of ⊕ when the input does not satisfy
the predicate p.
Optimization Theory of tails
In this section, we will give a theory for optimization of nested reductions describedwith tails, which is similar to that of inits.
The following two lemmas are well-known lemmas of tails [Bir87].

120

5. Domain-Specific Optimization for Skeleton Programs

Lemma 5.36 (Tails-map promotion). For function f , the following equation holds.
map (map f ) ◦ tails = tails ◦ map f
This lemma gives us a way to promote the application of function f throughtails. The number of applications of function f on the left hand side is n(n + 1)/2,
while that on the right hand side is n. So, transformation from the left hand side
to the right hand side improves the efficiency. Also, the lemma enables us to ignore
a function g in the generate-and-test specification ([f, ⊕]) ◦ map ([g, ⊗]) ◦ filter p ◦tails,
since we can replace g with the identity function id and apply g to the input of tails
beforehand (of course, we need some tricks to through filter p).
Lemma 5.37 (Scanr). For any associative binary operator ⊕, the following equation
holds.
scanr (⊕) = map (reduce (⊕)) ◦ tails
This lemma gives us the relation between computations of suffix sums by scanr
and by nested reductions with tails. It means that we can use scanr to computenested reductions if the outer reduction is id = ([[·], ++]), which results in the smaller
number of uses of the operator ⊕.
The following theorem gives efficient parallel implementation of nested reductions
for tails when two reductions have distributivity. One of the most famous problems
for which this theorem is applicable is maximum tail-segment sum problem (alsoknown as maximum suffix sum problem), which is an instance of maximum marking
problems [SHTO00, Bir01].
Theorem 5.38 (Maximum tail-segment sum). Provided that ⊕ is associative, and⊗ is associative and right-distributive over ⊕, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ tails = π1 ◦ ([pair , ⊛])where (t1 , s1 ) ⊛ (t2 , s2 ) = ((t1 ⊗ s2 ) ⊕ t2 , s1 ⊗ s2 )pair a = (a, a)
Proof. Similar to the proof of Theorem 5.33.
Similar to the previous theorem, the resulting program (the right hand side) of
the theorem uses only one reduction with the new operator ⊛, while the original
program (the left hand side) uses two nested reductions with tails. The cost of the
new operator is proportional to the cost of operators in the original reductions. So,
the resulting program is more efficient than the original program.
The new operator ⊛ is applied to tuples. The first element of a tuple is equal to
the result of the original program. The second element of the tuple is equal to the
reduction of the same input with operator ⊗, which is used to improve efficiency ofthe program by reusing the partial results effectively.
The following theorem extends Theorem 5.38, which allows filtering with thegiven predicate. It gives efficient parallel implementation of nested reductions for
tails when two reductions have distributivity and the predicate is relational.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

121

Theorem 5.39 (Maximum p-tail-segment sum). Provided that ⊕ is associative,⊗ is associative and right-distributive over ⊕, and predicate p is relational, thefollowing equation holds.
∗ ])reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ tails = π1 ◦ ([pentuple, 
where∗ (t2 , s2 , h2 , l2 , p2 ) =(t1 , s1 , h1 , l1 , p1 ) (t, s1 ⊗ s2 , h1 ≪ h2 , l1 ≫ l2 , p1 ∧ p2 ∧ p ([l1 , h2 ]))where i = (if p2 ∧ p ([l1 , h2 ]) then t1 ⊗ s2 else ı⊕ ) ⊕ t2
pentuple a = (a, a, a, a, T )Proof. Similar to the proof of Theorem 5.34.
The resulting program (the right hand side) of the theorem uses only one re∗ , while the original program (the left hand side)
duction with the new operator uses two nested reductions with tails. The cost of the new operator is proportional
to the cost of operators in the original reductions and the application of p. So, theresulting program is more efficient than the original program.
∗ is applied to pentuples. The first element of a pentuple is
The new operator equal to the result of the original program. The second element of the pentuple isequal to the reduction of the same input with operator ⊗, which is used to improve
efficiency of the program by reusing the partial results effectively. The third andforth elements are the edge elements of the input. Those edge elements are used tocheck whether results from two recursions in divide-and-conquer computation canbe connected to make a better solution. Since the predicate p is relational, we cancheck the connectability by using only elements on the edge. The fifth element is a
Boolean value that is the result of p applied to the input.
Here, we can reduce the size of pentuples by eliminating the fifth element (itcorresponds to p (x)) for simplicity, when we3 introduce an assumption that ı⊗ isthe zero of ⊗, i.e. a ⊗ ı⊕ = ı⊕ .
Corollary 5.40 (Maximum p-tail-segment sum (simplified)). Provided that ⊕ is
associative, ⊗ is associative and right-distributive over ⊕, the identity ı⊕ is the zeroof ⊗, and predicate p is relational, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ tails = π1 ◦ ([quadruple, ⊠
⊞])
where
(t1 , s1 , h1 , l1 ) ⊠⊞ (t2 , s2 , h2 , l2 ) = ((t1 ⊗ s2 )l1 ,h2 ⊕ t2 , (s1 ⊗ s2 )l1 ,h2 , h1 ≪ h2 , l1 ≫ l2 )
quadruple a = (a, a, a, a)(a)l,h = if p ([l, h]) then a else ı⊕
Proof. Similar to the proof of Corollary 5.35.
The resulting program (the right hand side) uses the new reduction operator ⊠⊞that is applied on quadruples. The difference from the result of Theorem 5.39 is that122

5. Domain-Specific Optimization for Skeleton Programs

the second element becomes the identity ı⊕ of ⊕ when the input does not satisfy
the predicate p.
A sufficient condition of the assumption that ı⊕ is the zero of ⊗ is that ⊗distributes over ⊕. So, we will use this condition in implementation of the library.
Optimization Theory of segs
In this section, we will give a theory for optimization of nested reductions describedwith segs.
The following lemma is well-known lemma of segs [Bir87].
Lemma 5.41 (Segs-map promotion). For function f , the following equation holds.
map (map f ) ◦ segs = segs ◦ map f
The lemma gives us a way to promote the application of function f through segs.
The number of applications of function f on the left hand side is O(n3 ), while thaton the right hand side is n. So, transformation from the left hand side to the righthand side improves the efficiency. Also, the lemma enables us to ignore a function gin the generate-and-test specification ([f, ⊕]) ◦ map ([g, ⊗]) ◦ filter p ◦segs, since we canreplace g with the identity function id and apply g to the input of segs beforehand(of course, we need some tricks to through filter p).
The following theorem gives efficient parallel implementation of nested reductions
for segs when two reductions have distributivity. One of the most famous problems
for which this theorem is applicable is maximum segment sum problem [Bir87],
which is an instance of maximum marking problems [SHTO00, Bir01].
Theorem 5.42 (Maximum segment sum). Provided that ⊕ is associative and commutative, and ⊗ is associative and distributive over ⊕, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ segs = π1 ◦ ([quadruple, ⊚])where(m1 , t1 , i1 , s1 ) ⊚ (m2 , t2 , i2 , s2 ) =(m1 ⊕ m2 ⊕ (t1 ⊗ i2 ), (t1 ⊗ s2 ) ⊕ t2 , i1 ⊕ (s1 ⊗ i2 ), s1 ⊗ s2 )quadruple a = (a, a, a, a)
Proof. We can prove the theorem by the following calculation.
LHS{ LHS }
reduce (⊕) ◦ map (reduce (⊗)) ◦ segs
= { definition of segs }reduce (⊕) ◦ map (reduce (⊗)) ◦ reduce (++) ◦ map inits ◦ tails= { promotion of map }reduce (⊕) ◦ reduce (++) ◦ map (map (reduce (⊗))) ◦ map inits ◦ tails= { promotion of reduce }reduce (⊕) ◦ map (reduce (⊕)) ◦ map (map (reduce (⊗))) ◦ map inits ◦ tails=

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements
=
=
=
=
=
=

123

{ distributivity of map }reduce (⊕) ◦ map ((reduce (⊕)) ◦(map (reduce (⊗))) ◦ inits) ◦ tails{ Theorem 5.33 }reduce (⊕) ◦ map (π1 ◦ ([pair , ⊙])) ◦ tails{ making ⊕′ so that (a1 , b1 ) ⊕′ (a2 , b2 ) = (a1 ⊕ a2 , b1 ⊕ b2 ), Lemma 5.36 }π1 ◦ reduce (⊕′ ) ◦ map (reduce (⊙)) ◦ tails ◦ map pair
{ Theorem 5.38 (commutativity of ⊕ guarantees distributivity of ⊙ over ⊕′ )}π1 ◦ π1 ◦ ([pair , ⊛]) ◦map pair{ fusing two π1 s, and two pair s }π1 ◦ ([quadruple, ⊚]){ RHS }
RHS

Definition of ⊛ in the above calculation is given as follows.
((m1 , t1 ), (i1 , s1 )) ⊛ ((m2 , t2 ), (i2 , s2 ))= { definition of ⊛ in Theorem 5.38 }(((m1 , t1 ) ⊙ (i2 , s2 )) ⊕′ (m2 , t2 ), (i1 , s1 ) ⊙ (i2 , s2 ))= { definition of ⊙ in Theorem 5.33 }((m1 ⊕ (t1 ⊗ i2 ), t1 ⊗ s2 ) ⊕′ (m2 , t2 ), (i1 ⊕ (s1 ⊗ i − 2), s1 ⊗ s2 ))= { definition of ⊕′ shown above }((m1 ⊕ (t1 ⊗ i2 ) ⊕ m2 , (t1 ⊗ s2 ) ⊕ t2 ), (i1 ⊕ (s1 ⊗ i − 2), s1 ⊗ s2 ))Flattening the nested pair into quadruple, we get the definition of ⊚.
The resulting program (the right hand side) of the theorem uses only one reduction with the new operator ⊚, while the original program (the left hand side) uses
two nested reductions with segs. The cost of the new operator is proportional tothe cost of operators in the original reductions. So, the resulting program is more
efficient than the original program.
The new operator ⊚ is applied to quadruples. The first element of a tuple isequal to the result of the original program. The second and the third elements are
results of the original nested reductions with tails and inits. The last element is equal
to the reduction of the same input with operator ⊗. These extra elements are used
to improve efficiency of the program by reusing the partial results effectively.
The following theorem extends Theorem 5.42, which allows filtering with thegiven predicate. It gives efficient parallel implementation of nested reductions forsegs when two reductions have distributivity and the predicate is relational. Derivation of efficient sequential implementations for those programs with filtering is shownin [Zan92, Jeu93].

124

5. Domain-Specific Optimization for Skeleton Programs

Theorem 5.43 (Maximum p-segment sum). Provided that ⊕ is associative andcommutative, ⊗ is associative and distributive over ⊕, the identity ı⊕ is the zero of⊗, and predicate p is relational, the following equation holds.
reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ segs = π1 ◦ ([hextuple, ])
where(m1 , t1 , i1 , s1 , h1 , l1 )  (m2 , t2 , i2 , s2 , h2 , l2 )= ( m1 ⊕ m2 ⊕ (t1 ⊗ i2 )l1 ,h2 , (t1 ⊗ s2 )l1 ,h2 ⊕ t2 , i1 ⊕ (s1 ⊗ i2 )l1 ,h2 ,(s1 ⊗ s2 )l1 ,h2 , h1 ≪ h2 , l1 ≫ l2 )hextuple a = (a, a, a, a, a, a)(a)l,h = if p ([l, h]) then a else ı⊕

Proof. We can prove the theorem by the following calculation.

=
=
=
=
=
=
=
=
=

LHS
{ LHS }
reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ segs
{ definition of segs }reduce (⊕) ◦ map (reduce (⊗)) ◦ filter p ◦ reduce (++) ◦ map inits ◦ tails{ promotion of filter , map and reduce }reduce (⊕) ◦ map (reduce (⊕)) ◦ map (reduce (⊗)) ◦ map (filter p) ◦ map inits ◦ tails{ map distributivity }reduce (⊕) ◦ map ((reduce (⊕)) ◦ (reduce (⊗)) ◦ (filter p) ◦ inits) ◦ tails{ Collorary 5.35 }reduce(π1 ◦ ([quadruple, ⊠])) ◦ tails (⊕) ◦ map′making ⊕ such that
(i1 , s1 , h1 , l1 ) ⊕′ (i2 , s2 , h2 , l2 ) = (i1 ⊕ i2 , s1 ⊕ s2 , h1 ≪ h2 , l1 ≫ l2 )π1 ◦ reduce (⊕′ ) ◦ map (reduce (⊠)) ◦ tails ◦ map quadruple{ Theorem 5.38 (⊠ distributes over ⊕′ ) }π1 ◦ π1 ◦ ([pair , ⊛]) ◦map quadruple{ fusing two π1 s, fusing pair and quadruple, removing duplicated parts }π1 ◦ ([hextuple, ]){ RHS }
RHS

Note that distributivity of ⊠ over ⊕′ is guaranteed for quadruples (i1 , s1 , h1 , l1 ) and(i2 , s2 , h2 , l2 ) of operands of ⊕′ when l1 = l2 . Condition l1 = l2 holds in the above
calculation since l1 and l2 are the last element of tail-segments of the same list.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

125

Definition of ⊛ in the above calculation is given as follows.
((m1 , t1 , h1 , l1 ), (i1 , s1 , k1 , n1 )) ⊛ ((m2 , t2 , h2 , l2 ), (i2 , s2 , k2 , n2 ))= { definition of ⊛ in Theorem 5.38 }(((m1 , t1 , h1 , l1 ) ⊠ (i2 , s2 , k2 , n2 )) ⊕′ (m2 , t2 , h2 , l2 ), (i1 , s1 , k1 , n1 ) ⊠ (i2 , s2 , k2 , n2 ))= { definition of ⊠ in Theorem 5.35 }((m1 ⊕ (t1 ⊗ i2 )l1 ,k2 , (t1 ⊗ s2 )l1 ,k2 , h1 ≪ k2 , l1 ≫ n2 ) ⊕′ (m2 , t2 , h2 , l2 ),(i1 ⊕ (s1 ⊗ i2 )n1 ,k2 , (s1 ⊗ s2 )n1 ,k2 , k1 ≪ k2 , n1 ≫ n2 )= { definition of ⊕′ shown above }((m1 ⊕ m2 ⊕ (t1 ⊗ i2 )l1 ,k2 , (t1 ⊗ s2 )l1 ,k2 ⊕ t2 , h1 ≪ k2 ≪ h2 , l1 ≫ n2 ≫ l2 ),(i1 ⊕ (s1 ⊗ i2 )n1 ,k2 , (s1 ⊗ s2 )n1 ,k2 , k1 ≪ k2 , n1 ≫ n2 )If h1 = k1 , l1 = n1 , h2 = k2 and l2 = n2 , then we have h1 ≪ k2 ≪ h2 = k1 ≪ k2 andl1 ≫ n2 ≫ l2 = n1 ≫ n2 . So, for octuple ((m, t, h, l), (i, s, k, n)), we have invarianth = k and l = n. Using this invariant, we can eliminate k and n from octuples andwe get computation using hextuples. Finally, flattening the hextuples, we get thedefinition of ⊚.
The resulting program (the right hand side) of the theorem uses only one reduction with the new operator , while the original program (the left hand side) uses
two nested reductions with segs. The cost of the new operator is proportional tothe cost of operators in the original reductions. So, the resulting program is more
efficient than the original program.
The new operator  is applied to hextuples. The first element of a tuple is equal
to the result of the original program. The second and the third elements are resultsof the original program in which segs is replaced with tails and inits. The fourth
element is equal to the reduction of the same input with operator ⊗. The fifth andthe sixth elements are the edge elements of the input. Those edge elements are used
to check whether results from two recursions in divide-and-conquer computation
can be connected to make a better solution. Since the predicate p is relational,
we can check the connectability by using only elements on the edge. These extra
elements are used to improve efficiency of the program by reusing the partial results
effectively.
Optimization Theory of neighbors
In this section, we will give a theory for optimization of nested reductions describedwith neighbors.
The following theorem links the result of domain specific fusion optimization in
Section 5.2 to the world of nested reductions.

126

5. Domain-Specific Optimization for Skeleton Programs

Theorem 5.44 (Finite-window neighbors). The following equation holds. Also, we
can apply the domain-specific fusion optimization in Section 5.2 to the program on
the right hand side.
map f ◦ neighbors ′ ls rs = map′ f ′ ◦ zipshift ls rs
where f ′ (a−m , . . . , a−1 , a0 , a1 , . . . , an ) = f ([a−m , . . . , a−1 ], a0 , [a1 , . . . , an ])zipshift ls rs x =let z0= x
z1= shift≪ r1 z0
z2= shift≪ r2 z1
..
.
zn
z−1

= shift≪ rn zn−1
= shift≫ l1 z0
..
.

z−m
= shift≪ lm z−(m−1)[lm , . . . , l1 ] = ls
[r1 , . . . , rn ] = rs
in zip z−m (zip z−(m−1) (. . . (zip z0 (. . . (zip zn−1 zn ) · · · )) · · · ))Proof. Skeleton compositions in the program on the right hand side has the target
pattern of the domain-specific fusion optimization in Section 5.2. Thus, we canapply the optimization to it.
The rest of the proof is to show the equation. We will show it by induction on
the input. We will use rules shown in Appendix C in the following proof.
The base case is trivial. Supposing the input is [a], we get z0 = [a], zi = ri (i ≥ 0),
and z−i = li (i ≥ 0), and thus the right hand side is f ([lm , . . . , l1 ], a, [r1 , . . . , rn ], whichis equal to the left hand side.
Next, we will show The induction case. To this end, we will first do some small
calculations, whose results will be used later.
map tail (tails (x ++ y))= { definition of tails }map tail (map (++head (tails y)) (tails x) ++ tails y)= { head ◦ tails = id by homomorphism fusion }map tail (map (++y) (tails x) ++ tails y)= { definition of map, and rule IV }map (++y) (map tail (tails x)) ++ map tail (tails y)
Similarly, we have the following equation.
map init (inits (x ++ y)) = map init (inits x) ++ map (x++) map init (inits y)
Using the above calculation, we have the following result for the left hand side. Some

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

127

where-clauses are omitted for readability.

=

=

=

=

(map f ◦ neighbors ′ ls rs) (x ++ y){ definition of neighbors ′ and neighbors }(map f ◦ map (taker m ◦ (ls++) × id × take n ◦ (++rs))◦zipP 3 ◦ ((map init ◦ inits)△id △(map tail ◦ tails))) (x ++ y)where zipP 3 (x, y, z) = zip x y zm = length ls
n = length rs{ the above calculation, and the definition of map }(map f ◦ map (taker m ◦ (ls++) × id × take n ◦ (++rs))◦ zipP 3 ◦ map (id × id × (++y)) ◦ ((map init ◦ inits)△id △(map tail ◦ tails))) x++ (map f ◦ map (taker m ◦ (ls++) × id × take n ◦ (++rs))◦ zipP 3 ◦ map ((x++) × id × id ) ◦ ((map init ◦ inits)△id △(map tail ◦ tails))) y{ fusion of zipP L3 and map, and associativity of ++ }(map f ◦ map (taker m ◦ (ls++) × id × take n ◦ (++(y ++ rs)))◦ zipP 3 ◦ ((map init ◦ inits)△id △(map tail ◦ tails))) x++ (map f ◦ map (taker m ◦ ((ls ++ x)++) × id × take n ◦ (++rs))◦ zipP 3 ◦ ((map init ◦ inits)△id △(map tail ◦ tails))) y{ take n ◦ take n = take n, taker m ◦ taker m = taker m, and folding }(map f ◦ neighbors ′ ls (take n (y ++ rs))) x++ (map f ◦ neighbors ′ (taker m (ls ++ x)) rs)) y

Similarly, we will calculate the right hand side. To this end, we first split the zi s
in the right hand side into zix s and ziy s. We use rule III to split them.

z0 = z0x ++ z0y = x ++ yz1 = z1x ++ z1y = shift≪ r1 (z0x ++ z0y ) = shift≪ (head z0y ) z0x ++ shift≪ r1 z0y
..
.
y
y
x
zn = znx ++ zny = shift≪ (head zn−1
) zn−1++ shift≪ rn zn−1yyx++ z−1 = shift≫ l1 z0x ++ shift≫ (last z0x ) z0
z−1 = z−1
..
.

y
y
x
x
x
z−m = z−m++ z−m= shift≫ lm z−(m−1)++ shift≫ (last z−(m−1)) z−(m−1)

Then, we show the following equation for i ≥ 0 by induction on i.
ziy = drop i (y ++ [r1 , . . . , ri ])128

5. Domain-Specific Optimization for Skeleton Programs

The base case i = 0 is trivial. The induction case is as follows.

=
=
=
=
=

ziy
{ definition }yshift≪ ri zi−1
{ induction hypothesis }shift≪ ri (drop (i − 1) (y ++ [r1 , . . . , ri−1 ])){ rule VI }tail (drop (i − 1) (y ++ [r1 , . . . , ri−1 ])) ++ [ri ]{ definition of drop }drop i (y ++ [r1 , . . . , ri−1 ]) ++ [ri ]{ y ++ [r1 , . . . , ri−1 ] has at least i elements }drop i (y ++ [r1 , . . . , ri−1 , ri ])Using the result we have the following result about zix .

=
=
=
=
s

zix
{ definition }yxshift≪ (head zi−1
) zi−1{ the above result }xshift≪ (head (drop (i − 1) (y ++ [r1 , . . . , ri−1 ]))) zi−1
{ definition of head , and (y ++ [r1 , . . . , ri−1 ]) has at least i elements. }xshift≪ (head (drop (i − 1) (y ++ rs))) zi−1
{ i is smaller than or equal to n }xshift≪ (head (drop (i − 1) (take n (y ++ rs)))) zi−1yx
We can show similar results on z−is and z−is.
Summarizing above calculation, we have the following result on the right handside.
(map′ f ′ ◦ zipshift ls rs) (x ++ y)= (map′ f ′ ◦ zipshift ls (take n (y ++ rs))) x++(map′ f ′ ◦ zipshift (taker m (ls ++ x)) rs) y

Therefore, using the induction hypothesis, we have shown the induction case for our
first equation.
Optimization Theory of Ts, Ls, Bs, Rs, TLs, TRs, BLs, and BRs
Similar to the functions inits and tails, those functions have promotion lemmas formap.
Lemma 5.45 (Map promotions for Ts, Ls, Bs, Rs, TLs, TRs, BLs, and BRs). For

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

129

any function f, the following equations hold.
map (map f ) ◦ Ts
map (map f ) ◦ Ls
map (map f ) ◦ Bs
map (map f ) ◦ Rs
map (map f ) ◦ TLs
map (map f ) ◦ TRs
map (map f ) ◦ BLs
map (map f ) ◦ BRs

=
=
=
=
=
=
=
=

Ts ◦ map f
Ls ◦ map f
Bs ◦ map f
Rs ◦ map f
TLs ◦ map f
TRs ◦ map f
BLs ◦ map f
BRs ◦ map f

Proof. We show the proof for Ts. The others can be proved similarly.

=

−
◦

=

map (map f ) ◦ Ts
{ definition of Ts }map (map f ) ◦ scan (−◦, ≫) ◦ map | · |{ definition of scan }map (map f ) ◦ ([| · |, ⊕′ , ⊗′ ]) ◦ map | · |where sx ⊕′ sy = sx−◦ mapr (zipwith (−◦) (bottom sx)) sy′sx ⊗ sy = sx mapc (zipwith (≫) (right sx)) sy{ definition of map, and fusion of homomorphism (Theorem 4.5) }([ map (map f ) ◦ | · | ◦ | · |, ⊕′ , ⊗′ ])where sx ⊕′ sy = sx−◦ mapr (zipwith (−◦) (bottom sx)) sy′sx ⊗ sy = sx mapc (zipwith (≫) (right sx)) sy{ map (map f ) ◦ | · | ◦ | · | = | · | ◦ | · | ◦ f }([| · | ◦ | · | ◦ f, ⊕′ , ⊗′ ])where sx ⊕′ sy = sx−◦ mapr (zipwith (−◦) (bottom sx)) sy′sx ⊗ sy = sx mapc (zipwith (≫) (right sx)) sy{ definition of map and scan }scan (−◦, ≫) ◦ map | · | ◦ map f{ definition of Ts }Ts ◦ map f
−
◦

=

−
◦

=

=
=

This lemma gives us a way to promote the application of function f through thegeneration functions. The number of applications of function f in the right handside is smaller than that in the left hand side.
The combination of the above lemma and the following lemma gives as a way topromote homomorphism through the generation functions.
Lemma 5.46 (Reducue promotions for Ts, Ls, Bs, Rs, TLs, TRs, BLs, and BRs).
For any binary operators which are associative and have the abide property, the130

5. Domain-Specific Optimization for Skeleton Programs

following equations hold.
map (reduce (⊖, ⊙)) ◦ Ts
map (reduce (⊖, ⊙)) ◦ Ls
map (reduce (⊖, ⊙)) ◦ Bs
map (reduce (⊖, ⊙)) ◦ Rs
map (reduce (⊖, ⊙)) ◦ TLs
map (reduce (⊖, ⊙)) ◦ TRs
map (reduce (⊖, ⊙)) ◦ BLs
map (reduce (⊖, ⊙)) ◦ BRs

=
=
=
=
=
=
=
=

scan (⊖, ≫)scan (≫, ⊙)scanr (⊖, ≪)scanr (≪, ⊙)scan (⊖, ⊙)scanr (≪, ⊙) ◦ scan (⊖, ≫)scanr (⊖, ≪) ◦ scan (≫, ⊙)scanr (⊖, ⊙)Proof. We show the proof for Ts. The others can be proved similarly.
=

−
◦

=

map (reduce (⊖, ⊙)) ◦ Ts
{ definition of Ts }map (reduce (⊖, ⊙)) ◦ scan (−◦, ≫) ◦ map | · |{ definition of scan }map (reduce (⊖, ⊙)) ◦ ([| · |, ⊕′ , ⊗′ ]) ◦ map | · |where sx ⊕′ sy = sx−◦ mapr (zipwith (−◦) (bottom sx)) sy′sx ⊗ sy = sx mapc (zipwith (≫) (right sx)) sy{ definition of map, and fusion of homomorphism (Theorem 4.5) }([ map (reduce (⊖, ⊙)) ◦ | · | ◦ | · |, ⊕′′ , ⊗′′ ])where sx ⊕′′ sy = sx−◦ mapr (zipwith (⊖) (bottom sx)) sysx ⊗′′ sy = sx mapc (zipwith (≫) (right sx)) sy{ map (reduce (⊖, ⊙)) ◦ | · | ◦ | · | = | · | }([| · |, ⊕′′ , ⊗′′ ])where sx ⊕′′ sy = sx−◦ mapr (zipwith (⊖) (bottom sx)) sy′′sx ⊗ sy = sx mapc (zipwith (≫) (right sx)) sy{ definition of scan }scan (⊖, ≫)
−
◦

=

−
◦

=

=

Optimization Theory of surrounds
In this section, we will give a theory for optimization of nested reductions describedwith surrounds.
The main result of optimization on surrounds is summarized as follows.
Theorem 5.47 (Surrounding). Let the function shrink be defined by homomorphisms as follows.
shrink = gc × hn × hs × he × hw × hne × hnw × hse × hsw
wherehn = ([gn , ⊕n , ⊗n ]) ,hs = ([gs , ⊕s , ⊗s ]) ,he= ([ge , ⊕e , ⊗e ])hw = ([gw , ⊕w , ⊗w ]) , hne = ([gne , ⊕ne , ⊗ne ]) , hnw = ([gnw , ⊕nw , ⊗nw ])hse = ([gse , ⊕se , ⊗se ]) , hsw = ([gsw , ⊕sw , ⊗sw ])5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

131

Here, ⊕X and ⊗X are extended to satisfy the following equations: NIL ⊕X x = x,x ⊕X NIL = x, NIL ⊗X x = x, and x ⊗X NIL = x.
Then, there exist a projection function proj and operators ⊕′f , ⊗′f , ⊕′r , and ⊗′r ,and the following equation holds.
map f ◦ map shrink ◦ surrounds =map (f ◦ proj ) ◦ scanr(⊕′r , ⊗′r ) ◦ map fr ′ ◦ scan(⊕′f , ⊗′f ) ◦ map ff ′Here, the complexity of each of the operators ⊕′f , ⊗′f , ⊕′r , and ⊗′r is bounded by thelargest of ⊕X and ⊗X .
Proof. The theorem is proved by the promotion of map shrink with extending the tuples.
The first step is to promote map shrink through scanr(⊕r , ⊗r ) since the target programis written as follows:
map shrink ◦ surrounds = map shrink ◦ scanr(⊕r , ⊗r ) ◦ map fr ◦ scan(⊕f , ⊗f ) ◦ map ff .
To do it, we calculate the terms shrink (a ⊕r b) and shrink (a ⊗r b) .
shrink ((ca , na , sa , ea , wa , nea , nwa , sea , swa ) ⊕r (cb , nb , sb , eb , wb , neb , nwb , seb , swb ))={ Definition of ⊕r }◦ eb −◦ wb −◦ |cb |−◦ sb , ea , wa , nea , nwa , sea −◦ seb , swa −shrink (ca , na , sa −◦ swb )=
{ Definition of shrink }◦ |cb |−◦ sb ), he ea , hw wa , hne nea , hnw nwa ,(gc ca , hn na , hs (sa −◦ eb −◦ wb −hse (sea −◦ seb ), hsw (swa −◦ swb ))=
{ Definition of homomorphism }(gc ca , hn na , hs sa ⊕s gs cb ⊕s hs sb , he ea , hw wa , hne nea , hnw nwa ,hse sea ⊕se hse eb ⊕se hse seb , hsw swa ⊕sw hsw wb ⊕sw hsw swb )

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

shrink ((ca , na , sa , ea , wa , nea , nwa , sea , swa ) ⊗r (cb , nb , sb , eb , wb , neb , nwb , seb , swb ))={ Definition of ⊗r }shrink (ca , na , sa , ea |cb | eb , wa , nea nb neb , nwa , sea sb seb , swa )={ Definition of shrink }(gc ca , hn na , hs sa , he (ea |cb | eb ), hw wa ,hne (nea nb neb ), hnw nwa , hse (sea sb seb ), hsw swa )={ Definition of homomorphism }(gc ca , hn na , hs sa , he ea ⊗e ge cb ⊗e he eb , hw wa ,hne nea ⊗ne hne nb ⊗ne hne neb , hnw nwa , hse sea ⊗se hse sb ⊗se hse seb , hsw swa )Here, three different functions, namely gc , ge and gs , are applied to cb and ca . To makethe values ge cb and gs cb , we have to store always cb as well as gc cb that is the result of thefirst element. Similarly, n, s, e and w have applications of two different functions. Thus,

132

5. Domain-Specific Optimization for Skeleton Programs

we extend the tuple to store such extra values duplicating c, n, s, e and w as follows.

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

map shrink ◦ scanr(⊕r , ⊗r ) ◦ map fr = proj ◦ map shrink ′ ◦ scanr(⊕′r , ⊗′r ) ◦ map fr ′whereproj (c, n, s, e, w, ne, nw, se, sw, c′ , n′ , s′ , e′ , w′ ) = (c, n, s, e, w, ne, nw, se, sw)
shrink ′ = gc ×hn ×hs ×he ×hw ×hne ×hnw ×hse ×hsw ×id ×hne ×hse ×hse ×hsw
fr ′ (c, n, w, nw) = (c, n, NIL, NIL, w, NIL, nw, NIL, NIL, c, n, NIL, NIL, w)(ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊕r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )◦ swb ,
◦ wb′ −◦ seb , swa −◦ e′b −
◦ |c′b |−◦ sb , ea , wa , nea , nwa , sea −= (ca , na , sa −′′′′◦ |cb |−
◦ s′b , e′a , wa′ )c a , na , sa −
′
′
′
′
′
(ca , na , sa , ea , wa , nea , nwa , sea , swa , ca , na , sa , ea , wa )⊗r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )= (ca , na , sa , ea |c′b | eb , wa , nea n′b neb , nwa , sea s′b seb , swa ,c′a , n′a , s′a , e′a |c′b | e′b , wa′ )Then, we calculate the new terms shrink ′ (a ⊕′r b) and shrink ′ (a ⊗′r b), .

shrink ((ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊕′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )={ Definition of ⊕′r , shrink ′ and homomorphism }(gc ca , hn na , hs sa ⊕s gs c′b ⊕s hs sb , he ea , hw wa , hne nea , hnw nwa ,hse sea ⊕se hse e′b ⊕se hse seb , hsw swa ⊕sw hsw wb′ ⊕sw hsw swb ,c′a , hne n′a , hse s′a ⊕se gse c′b ⊕se hse s′b , he e′a , hw wa′ )={ Creating a new operator, Definition of shrink ′ }shrink (ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊕′′r shrink (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )
where(ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊕′′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )= (ca , na , sa ⊕s gs c′b ⊕s sb , ea , wa , nea , nwa ,sea ⊕se e′b ⊕se seb , swa ⊕sw wb′ ⊕sw swb ,c′a , n′a , s′a ⊕se gse c′b ⊕se s′b , e′a , wa′ )shrink ((ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊗′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )={ Definition of ⊗′ , shrink ′ and homomorphism }(gc ca , hn na , hs sa , he ea ⊗e ge c′b ⊗e he eb , hw wa ,hne nea ⊗ne hne n′b ⊗ne hne neb , hnw nwa , hse sea ⊗se hse s′b ⊗se hse seb , hsw swa ,ca , hne n′a , hse s′a , hse e′a ⊗se gse c′b ⊗se he e′b , hsw wa′ )={ Creating a new operator, Definition of shrink ′ }shrink (ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊗′′ shrink (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )
where(ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊗′′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )= (ca , na , sa , ea ⊗e ge c′b ⊗e eb , wa , nea ⊗ne n′b ⊗ne neb , nwa , sea ⊗se s′b ⊗se seb , swa ,ca , n′a , s′a , e′a ⊗se gse c′b ⊗se e′b , wa′ )5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

133

Using the above operators, we complete the first promotion:
map shrink ′ ◦ scanr(⊕′r , ⊗′r ) = scanr(⊕′′r , ⊗′′r ) ◦ map shrink ′ .
The second step is the promotion of map shrink ′ through map fr ′ .
shrink ′ (fr ′ (c, n, w, nw))=
{ Definition of shrink ′ and fr ′ }(gc c, hn n, hs NIL, he NIL, hw w, hne NILhnw nw, hse NIL, hsw NIL,c, hne n, hse NIL, hse NIL, hsw w)={ Application of hX to NIL results in NIL }(gc c, hn n, NIL, NIL, hw w, NILhnw nw, NIL, NIL, c, hne n, NIL, NIL, hsw w)
Here, each of the w and n has two applications of different functions. Thus, we extend
the tuple used in the first stage calculation of surrounds duplicating w and n as follows.

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

map shrink ′ ◦ map fr ′ ◦ scan(⊕f , ⊗f ) ◦ map ff= map shrink ′ ◦ map fr ′′ ◦ scan(⊕′f , ⊗′f ) ◦ map ff ′where
fr ′ (c, n, w, nw, n′ , w′ , n′′ , w′′ ) = (c, n, NIL, NIL, w, NIL, nw, NIL, NIL, c, n′ , NIL, NIL, w′ )ff ′ a = (a, NIL, NIL, NIL, NIL, NIL, NIL, NIL)
(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊕′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )◦ |ca |−
◦ n′′b , wb′′ )◦ |ca |−
◦ n′b , wb′ , n′′a −◦ nwb , n′a −◦ wa′′ −◦ |ca |−
◦ nb , wb , nwa −= (cb , na −(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊗′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )= (cb , nb , wa |ca | wb , nwa n′′a nwb , n′b , wa′ |ca | wb′ , n′′b , wa′′ |ca | wb′′ )The extra duplication of w and n is used later. Now, we do the promotion of map shrink ′through map fr ′′ instead of map fr ′ .
shrink ′ (fr ′′ (c, n, w, nw, n′ , w′ , n′′ , w′′ ))={ Definition of shrink ′ and fr ′′ }(gc c, hn n, hs NIL, he NIL, hw w, hne NILhnw nw, hse NIL, hsw NIL,c, hne n′ , hse NIL, hse NIL, hsw w′ )={ Application of hX to NIL results in NIL }(gc c, hn n, NIL, NIL, hw w, NILhnw nw, NIL, NIL, c, hne n′ , NIL, NIL, hsw w′ )={ Creating a new function }fr ′′′ (shrink ′′ (c, n, w, nw, n′ , w′ , n′′ , w′′ ))where
fr ′′′ (c, n, w, nw, n′ , w′ , n′′ , w′′ )= (gc c, n, NIL, NIL, w, NILnw, NIL, NIL, c, n′ , NIL, NIL, w′ )shrink ′′ = id × hn × hw × hnw × hne × hsw × hnw × hnw
Thus, map shrink ′ ◦ map fr ′′ = map fr ′′′ ◦ map shrink ′′ .
The first step of the rest of the promotion is the calculation of shrink ′′ (a ⊕′f b) as

134

5. Domain-Specific Optimization for Skeleton Programs

follows.
shrink ′′ ((ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊕′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ ))={ Definition of ⊕′f and shrink ′′ }◦ nwb ),
◦ wa −◦ |ca |−
◦ nb ), hw wb , hnw (nwa −(cb , hn (na −◦ |ca |−
◦ n′′b ), hnw wb′′ )◦ |ca |−
◦ n′b ), hsw wb′ , hnw (n′′a −hne (n′a −={ Definition of homomorphism }(cb , hn na ⊕n gn ca ⊕n hn nb , hw wb , hnw nwa ⊕nw hnw wa′′ ⊕nw hnw nwb ,hne n′a ⊕ne gne ca ⊕ne hne n′b , hsw wb′ , hnw n′′a ⊕nw gnw ca ⊕nw hnw n′′b , hnw wb′′ )={ Creating a new operator }shrink ′′ (ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ⊕′′f shrink ′′ (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )where(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊕′′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )= (cb , na ⊕n gn ca ⊕n nb , wb , nwa ⊕nw wa′′ ⊕nw nwb ,n′a ⊕ne gne ca ⊕ne n′b , wb′ , n′′a ⊕nw gnw ca ⊕nw n′′b , wb′′ )Similarly, the calculation of shrink ′′ (a ⊗′f b) is as follows.
shrink ′′ ((ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊗′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ ))={ Definition of ⊗′ , shrink ′′ and homomorphism }(cb , hn nb , hw wa ⊗w gw ca ⊗w hn wb , hnw nwa ⊗nw hnw n′′a ⊗nw hnw nwb ,hne n′b , hsw wa′ ⊗sw gsw ca ⊗sw hsw wb′ , hnw n′′b , hnw wa′′ ⊗nw gnw ca ⊗nw hnw wb′′ )={ Creating a new operator }shrink ′′ (ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ⊗′′ shrink ′′ (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )where(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊗′′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )= (cb , nb , wa ⊗w gw ca ⊗w wb , nwa ⊗nw n′′a ⊗nw nwb ,n′b , wa′ ⊗sw gsw ca ⊗sw wb′ , n′′b , wa′′ ⊗nw gnw ca ⊗nw wb′′ )The final step is fusion of shrink ′′ and ff ′ .
shrink ′′ (ff ′ c)=
{ Definition of shrink ′′ and ff ′ }(c, hn NIL, hw NIL, hnw NIL, hne NIL, hsw NIL, hnw NIL, hnw NIL)=
{ Application of hX to NIL results in NIL }(c, NIL, NIL, NIL, NIL, NIL, NIL, NIL)=
{ Definition of ff ′ }
ff ′ cGathering the above results, the optimization is done as follows.

5.3. Nested Reductions: Involving an Infinite Number of Neighbor Elements

135

map shrink ◦ surrounds = proj ◦ scanr(⊕′′r , ⊗′′r ) ◦ map fr ′′′ ◦ scan(⊕′′f , ⊗′′f ) ◦ map ff ′where
ff ′ a = (a, NIL, NIL, NIL, NIL, NIL, NIL, NIL)
(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊕′′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )= (cb , na ⊕n gn ca ⊕n nb , wb , nwa ⊕nw wa′′ ⊕nw nwb ,n′a ⊕ne gne ca ⊕ne n′b , wb′ , n′′a ⊕nw gnw ca ⊕nw n′′b , wb′′ )(ca , na , wa , nwa , n′a , wa′ , n′′a , wa′′ ) ⊗′′f (cb , nb , wb , nwb , n′b , wb′ , n′′b , wb′′ )= (cb , nb , wa ⊗w gw ca ⊗w wb , nwa ⊗nw n′′a ⊗nw nwb ,n′b , wa′ ⊗sw gsw ca ⊗sw wb′ , n′′b , wa′′ ⊗nw gnw ca ⊗nw wb′′ )fr ′′′ (c, n, w, nw, n′ , w′ , n′′ , w′′ )= (gc c, n, NIL, NIL, w, NILnw, NIL, NIL, c, n′ , NIL, NIL, w′ )(ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊕′′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )= (ca , na , sa , ea ⊗e ge c′b ⊗e eb , wa ,nea ⊗ne n′b ⊗ne neb , nwa , sea ⊗se e′b ⊗se seb , swa ,ca , n′a , s′a , e′a ⊗se gse c′b ⊗se e′b , wa′ )(ca , na , sa , ea , wa , nea , nwa , sea , swa , c′a , n′a , s′a , e′a , wa′ )⊗′′r (cb , nb , sb , eb , wb , neb , nwb , seb , swb , c′b , n′b , s′b , e′b , wb′ )= (ca , na , sa , ea ⊗e ge c′b ⊗e eb , wa ,nea ⊗ne n′b ⊗ne neb , nwa , sea ⊗se s′b ⊗se seb , swa ,ca , n′a , s′a , e′a ⊗se gse c′b ⊗se e′b , wa′ )The resulting program uses O(n2 ) operations for a two-dimensional array ofn × n, while the original general form uses p
O(n4 ) operations. The parallel complexity of the resulting program is O((n2 /P + n2 /P log P )T(⊕X ,⊗X ) ) for P processors,
provided that the computational complexity of ⊕X and ⊗X in the homomorphisms
are T(⊕X ,⊗X ) .
All the examples shown in the previous section are described in the left handside of the theorem. Thus,p we can apply this theorem to all of them, and they are
executed in O(n2 /P + n2 /P log P ) complexity using the skeletons.
It is easily seen that the duplication of elements in the tuple during the derivation is not necessary if some of the functions hX (X = {n, s, e, w, ne, nw, se, sw})
have the same operators. For example, if hn and hne have the same operator, i.e.
hn = ([gne , ⊕ne , ⊗n ]) and hne = ([gne , ⊕ne , ⊗ne ]) , then we can use hn n instead ofhne n because hn n = hne n when width n = 1 . Thus, in this case we can remove
the duplication of n for hne n in the tuple. Note that ⊗n does not need to be thesame of ⊗ne because it is not used.
Corollary 5.48 (Reduction of the tuple). If the neighboring functions of hX (X ∈{n, s, e, w, ne, nw, se, sw}) have the same operators, then, the size of the tuples usedin the final program can be reduced. The minimum size is nine.

136

5. Domain-Specific Optimization for Skeleton Programs

Finally, we note that we may perform more optimizations by using the shifting of
the edges instead of butterfly computations for the global computations of scan andscanr, provided that the operators influence only
p a fixed number of elements. This2
leads to the parallel complexity of O((n /P + n2 /P )T(⊕X ,⊗X ) ) for P processors.
Optimization Theory of rects ′We will show a lemma and a theorem that optimizes nested reductions with rects ′ .
The theorem is the generalized result of the derivation in Section 4.3.4.
Lemma 5.49 (Map promotion for rects ′ ). For any function f , the following equation
holds.
map (map f ) ◦ rects ′ = rects ′ ◦ map f
Proof. This lemma is proved by the following calculation.

=
=
=
=
=
=

map (map f ) ◦ rects ′{ definition of rects ′ }map (map f ) ◦ flatten ◦ map (TLs) ◦ BRs{ promotion of map f through flatten }flatten ◦ map (map (map f )) ◦ map (TLs) ◦ BRs{ map-map fusion }flatten ◦ map (map (map f ) ◦ TLs) ◦ BRs{ map promotion through TLs }flatten ◦ map (TLs ◦ map f ) ◦ BRs{ the inverse of map-map fusion, and map promotion through BRs }flatten ◦ map TLs ◦ BRs ◦ map f{ definition of rects ′ }rects ′ ◦ map f

Theorem 5.50. For any associative, commutative binary operators ⊕ and ⊗ withidentities such that ⊗ distributes over ⊕, the following equation holds.
reduce (⊕, ⊕) ◦ map (reduce (⊗, ⊗)) ◦ rects ′ = π1 ◦ ([i11
1fi′′′ ,i11
1⊙′′i ,i11
1⊖′′i ])

Here, functions fi′′′ and the operators ⊙′′i and ⊖′′i are those given by generalizing theoperators ↑ and + in the derivation of Section 4.3.4 to ⊕ and ⊗.
Proof. The proof is given by generalizing the operators ↑ and + in the derivation ofSection 4.3.4 to ⊕ and ⊗.

5.4. Related Work

5.4

137

Related Work

Optimization of Skeleton Programs Involving Neighbor Elements
One of the simplest fusion optimizations on lists so far uses a general form called
cataJ [MKI+ 04]. This cataJ is actually a list homomorphism that has a function
applied to each element of the input list, and an associative binary operator used toperform reduction on elements. Thus, cataJ can describe any computation written
as composition of any number of map and at most one reduce (a skeleton to perform
reduction) at the last. In this sense, cataJ is a normal form of skeleton programs ofsuch compositions.
Hu et al. [HIT02] proposed a general fusion optimization using a general form
called accumulate and a set of fusion rules. The accumulate can describe skeleton
scan′ , which calculates an accumulation of the input list with an associative binary
operator, as well as map and reduce. So, it can be a normal form of skeleton programs described with compositions of these skeletons. Although accumulate candescribe also shift≫ and shift≪, it causes some overheads due to lack of considerationof elements on edges. Main overheads are as follows: (1) extensions of elements
for uniform manipulation by the associative binary operator, and (2) logarithmic
steps of inter-processor communications for general implementation of accumulation. Thus, we need to consider a specific fusion optimization, i.e. a normal form,
fusion rules and efficient implementation.
Our proposed normal form extends these fusions optimizations with consideration of elements on edges introduced by shift≫ and shift≪. The normal form separates
computation of elements on breaks from that of center part, so that it does not introduce the overheads that accumulate causes.
Grelck et al. [GS06] proposed an optimization based on fusion of a general skeleton called WITH-loop. Since their WITH-loop are based on index accessing, their
method can deal with shifting operation such as shift≫ and shift≪. Also, their fusion
can handle nesting use of skeletons. However, skeletons that perform accumulation
or reduction, in which a region of elements required in computation of an element
varies, cannot be described with WITH-loop. Our skeletons can handle such computation owing to the homomorphism-based design.
Generalized scan (gen scan) proposed by Fischer et al. [FPS01] can perform accumulation that involves neighbor elements. We can describe shifting operations suchas shift≫ and shift≪ with their gen scan. We think program that perform accumulation after combinations of map, zip, shift≫ and shift≪ can be described by gen scan.
However, since the condition for parallel implementation of gen scan is very complicated, formalization of fusion rules of gen scan is very difficult. Our skeletons areeasier to build fusion rules owing to their uniform design based on homomorphism.

138

5. Domain-Specific Optimization for Skeleton Programs

Maximum Marking Problem
Maximum marking problem is one of optimization problems on sequences. The objective of maximum marking problem is to find a marking of the input sequencethat gives the maximum weight sum. We can produce many instances of maximum
marking problem by changing the rule of marking on the input sequence. For example, maximum prefix sum problem , maximum suffix sum problem, and maximum
segment sum problem are instances of maximum marking problem.
Derivation of efficient sequential programs for maximum marking problems wasstudied by Sasano et al. [SHTO00,SHT01] and Bird [Bir01]. They showed systematicconstruction of efficient sequential programs when rules of marking are given asrecursive functions of a certain class. Since marking of a prefix/suffix/segment
is given as a recursive function of the class, an efficient sequential program for
maximum prefix/suffix/segment problem is obtained systematically by their results.
Longest Segment Problems
The objective of longest segment problems is to find the longest segment that satisfies a given predicate. Derivation of efficient sequential programs for longest segment problems were studied by Zatema [Zan92], Jeuring [Jeu93] and Zhao [Zha02].
Zatema [Zan92] and Jeuring [Jeu93] showed derivation of efficient sequential programs for various predicates. Zhao [Zha02] studied derivation of efficient sequential
programs for predicates constructed as a combination of primitive predicates. She
applied it for data mining to make a querying system that supports efficient queryingof combined predicates.
Nested Reductions on Two-dimensional Arrays
SKiPPER [SG02] is a skeleton-based parallel programming environment for realtime image processing. It has skeletons specialized for image processing, while we
use general skeletons on two-dimensional arrays. Thus, a program developed with
SKiPPER may be faster than our skeleton programs in the domain of image processing. Our skeletons deal with wider range of problems, though.

Chapter 6
Implementation of Skeletons andOptimizations
In this chapter, we will show parallel implementations of our designed parallel skeletons for lists and two-dimensional arrays, simple systems for the fusion optimizations, and a library with optimization capabilities for nested reductions.
First, we will show to implement the designed skeletons in parallel for distributed
parallel environments. The implementation relies on the parallelism of the homomorphism. We will report some experiment results.
Next, we will explain small systems for fusion optimizations designed in Chapters 4 and 5, and show experiment results of the optimizations. 1 Finally, we willpropose a general design of libraries with optimization capabilities based on optimization theorems for domain-specific optimizations, and show implementation of alibrary for nested reductions in Fortress [ACH+ 08], of which principle for optimization have been developed in Chapter 5.

6.1

Implementation of Parallel Skeletons on Lists

We briefly view the parallel implementation of the skeletons. In the following, we
consider the simplest case: there are n processors and each of them has one elementof the list of length n .
The implementation of map and zipwith is quite simple as shown in Figure 6.1.
The figure shows that the list [1, 2, 3, 4, 5, 6, 7, 8] is distributively stored in eightprocessors P0 through P7. Each element of the resulting list can be calculated
independently on each processor. Thus, the parallel computational time complexityof map is O(Tf ), where Tf is the complexity of the given function. Similarly, thecomplexity of zipwith is O(Tf ).
The calculation of reduce is performed by tree-like computation shown in Figure
6.2. Each stage of the computation is as follows: active processors of the evennumber positions receive the values from corresponding active processors of the odd
number positions, then they apply the operator to the values, and finally they store

140

time

6. Implementation of Skeletons and Optimizations
2

4

6

8

10

12

14

16

dbl

dbl

dbl

dbl

dbl

dbl

dbl

dbl

1

2

3

4

5

6

7

8

P0

P1

P2

P3

P4

P5

P6

P7

Step.1pro essors

Figure 6.1. Parallel Computation of map dbl [1, 2, 3, 4, 5, 6, 7, 8]

time

36
+

Step.3

10
+

26
+

3
+

7
+

Step.2

11+

15+

Step.1

1

2

3

4

5

6

7

8

P0

P1

P2

P3

P4

P5

P6

P7

pro essors

Figure 6.2. Parallel Computation of reduce (+) [1, 2, 3, 4, 5, 6, 7, 8]

the results for the nest stage. The processors of the odd number positions become
idle after sending their values. One stage of the tree-like computation is done inO(T⊕ ) time in parallel, where T⊕ is the complexity of the operator. Repeating thestages the communication forms a tree, and its height is O(log(n)) . Thus, theparallel computational time complexity of reduce is O(log(n)T⊕ ) .
The implementation of scan is similar to that of reduce. It uses the butterfly
computation in stead of the tree-like computation. It is shown in Figure 6.3. Here,each processor stores the pair of the results of applying reduce to the preceding
elements and to all elements in the segment considered, respectively. Thus, theparallel computational complexity of scan is O(log(n)T⊕ ) .

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays
1

3

1

6

1

10

1

15

1

21

1

28

1

141

36

1

1

time

Step.4(1,36)(3,36)(6,36)(10,36)(15,36)(21,36)(28,36)(36,36)

(+;

(+;

(+;

(+;

(+; +)(+; +)(+; +)(+; +)



)



)



)



)

Step.3(1,10)(3,10)(6,10)(10,10)(5,26)(11,26)(18,26)(26,26)(+;(+;(+; +)(+; +)(+;(+;(+; +)(+; +)



)

(1,3)
(+;



)



)

(3,3)

(3,7)

(+; +)

(+;



)



)



)

Step.2(7,7)(5,11)(11,11)(7,15)(15,15)(+; +)(+;(+; +)(+;(+; +)

))

Step.1

(1,1)

(2,2)

(3,3)

(4,4)

(5,5)

(6,6)

(7,7)

(8,8)

pair

pair

pair

pair

pair

pair

pair

pair
Step.0

1

2

3

4

5

6

7

8

P0

P1

P2

P3

P4

P5

P6

P7

pro essors

Figure 6.3. Parallel Computation of scan (+) [1, 2, 3, 4, 5, 6, 7, 8]

6.2

Implementation of Parallel Skeletons on Twodimensional Arrays

In this section, we will report efficient parallel implementations of parallel skeletons for two-dimensional data structures on distributed parallel machines, such as
PC clusters. We will show some experimental results on the parallel skeletons implemented on C++ with MPI [SO98, GLS99], to show programs described withskeletons can be executed efficiently in parallel. We used MPI to implement lowerlevel communication so that we can use skeletons on distributed-memory machines.
Actually, we have alternatives to implement skeletons. For example, we may use
OpenMP [CDK+ 01, CJvdP07] to provide implementation of skeletons for sharedmemory machines.
First, we will show implementations of map, zipwith, and reduce, which have
simple implementations. Then, we will give implementation of scan, which has more
involved implementation. After that, we will report some experiment results.

142

6.2.1

6. Implementation of Skeletons and Optimizations

Implementation of Simple Parallel Skeletons

The basic parallel skeletons map, zipwith, and reduce can be efficiently implemented
on distributed memory systems. To illustrate this, we separate computations of askeleton into two parts: local computations within a processor and global computations crossing processors. To this end, we will introduce two functions that represent
distribution and gathering of arrays.
The function dist abstracts distribution of a two-dimensional array to p × q
processors, and is defined as follows.
dist p q x = (flatten ◦ map (grp c n) ◦ grp r m) x
where m = ⌈height x/p⌉, n = ⌈width x/q⌉

The function grp r divides the given array into sub-arrays of height k, except for thelast one. It is defined as follows.
grp r k (x−◦ y) = |x|−
◦ (grp r k y) while height x = k
grp r k x
= |x|
when height x ≤ kNote that our arrays can change its structure so that x of x −◦ y has the height ofk when the height of x −◦ y is greater than k. The function grp c divides the input
array similarly in the alternative direction, and is defined similarly.
The distribution dist p q x means that the two-dimensional array x will be divided
into p × q subarrays (i.e. x is divided into p subarrays in vertical direction, theneach subarray is divided into q subarrays in horizontal direction), and each subarrayis distributed to each processor.
The function gather, the inverse operator of dist, abstracts gathering of twodimensional arrays distributed to the processors into a two-dimensional array on
the root processor.
−◦

gather = reduce (−◦, )Although definitions of these functions may seem complicated, actual operations
are simple as illustrated in Figure 6.4. What is significant here is that these functions
satisfy the relation of id = gather ◦ dist p q.
Now, we will proceed to implementations of skeletons. Here, we will separate
computations of a skeleton into two parts: local computations within a processor
and global computations crossing processors.
For map skeleton, we can separate its computation as follows.
{ id = gather ◦ dist p q }map f ◦ gather ◦ dist p q={ Definition of gather }map f ◦ reduce (−◦, ) ◦ dist p q={ Promotion of map f }reduce (−◦, ) ◦ map (map f ) ◦ dist p q={ Definition of gather }gather ◦ map (map f ) ◦ dist p q
−
◦

−
◦

map f =

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays

X01 X11 X21
X00 X10 X20

dist p q

X01 X11 X21

gather

X00 X10 X20

143

Figure 6.4. An image of distribution function dist p q and gathering function gather
with p = 2 and q = 3. Each rectangle corresponds to each processor; Xij represents
a subarray.

The last formula indicates that we can compute map f by distributing a twodimensional array of the argument to the processors by dist p q, applying map f toeach local array independently on each processor, and finally gathering the results
onto the root processor by gather. Thus, for a two-dimensional array of n × n size
we can compute map f in O(Tf n2 /P ) parallel time, using P = pq processors andignoring the distribution and the collection, provided that the function f can becomputed in O(Tf ) time.
The implementation of zipwith is almost the same as map.
For reduce skeleton, we can separate its computation as follows.
{ id = gather ◦ dist p q }reduce (⊕, ⊗) ◦ gather ◦ dist p q={ Definition of gather }reduce (⊕, ⊗) ◦ reduce (−◦, ) ◦ dist p q={ Promotion of reduce (⊕, ⊗) }reduce (⊕, ⊗) ◦ map (reduce (⊕, ⊗)) ◦ dist p q−◦

reduce (⊕, ⊗) =The last formula indicates that we can compute reduce (⊕, ⊗) by distributing a
two-dimensional array of the argument to the processors by dist p q, applying
reduce (⊕, ⊗) to each local array independently on each processor, and finally reducing the results into the root processor by reduce (⊕, ⊗) described in the last
formula. From the property of Eq. (3.2), the last reduction over the results of allprocessors can be computed by using tree-like computation in column and row directions respectively like parallel computation of reduction on one-dimensional lists(see Chapter 2). Thus, for a two-dimensional array of n × n size we can compute
reduce (⊕, ⊗) in O(T⊕,⊗ (n2 /P + log P )) parallel time, using P = pq processors andignoring the distribution, provided that the binary operators ⊕ and ⊗ can be computed in O(T⊕,⊗ ) time. Note that we can also execute the tree-like computation inboth of column and row directions simultaneously, which results in the same cost.

6.2.2

Implementation of Scan Skeleton

In this section, we will show the parallel implementation of scan, as well as theimplementation of accumulate. We will first give the implementation of scan for144

6. Implementation of Skeletons and Optimizations

the simple case that each processor has one element of the two-dimensional array.
Then, we will give the general implementation of scan, in which each processor has
a subarray of the input rather than a single element. We will first show the general
implementation of accumulate using the simple case implementation of scan, andthen instantiate it for scan.
To formalize the simple case implementation of scan, we first extend the distributable homomorphism [Gor96] on lists to that on two-dimensional arrays. The
distributable homomorphism on lists is a special case of homomorphism, and can beused as the basis of parallel implementation of scan on lists. The distributable homomorphism on two-dimensional arrays is also a special case of the homomorphism,
as defined below.

−
◦

−
◦

DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (x−◦ y) = let x′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) x
y ′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) yin zipwith (⊞) x′ y ′ −◦ zipwith (⊞′ ) x′ y ′DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (x y) = let x′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) x
y ′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) yin zipwith (⊠) x′ y ′ zipwith (⊠′ ) x′ y ′′′DH2 (⊞, ⊞ , ⊠, ⊠ ) |a|= |a|

It is easily seen that the distributable homomorphism DH2 (⊞, ⊞′ , ⊠, ⊠′ ) is executedin parallel by the two-dimensional version of the butterfly computation as shown in
Figure 6.5. It is also clear that its computational complexity is O(T log(n)) for ann × n array, where T is the complexity of the operators.
Then, we give the theorem that shows the skeleton scan can be described interms of the distributable homomorphism; the skeleton scan can be implemented
using the butterfly computation of the distributable homomorphism.
Theorem 6.1 (Implementation of scan by distributable homomorphism). The skeleton scan is described in terms of distributable homomorphism as follows.
scan (⊕, ⊗) = map π1 ◦ DH2 (⊞, ⊞′ , ⊠, ⊠′ ) ◦ map p4
where(nwa , na , wa , ca ) ⊞ (nwb , nb , wb , cb ) = (nwa , na , wa ⊕ wb , ca ⊕ cb )(nwa , na , wa , ca ) ⊞′ (nwb , nb , wb , cb ) = (wa ⊕ nwb , ca ⊕ nb , wa ⊕ wb , ca ⊕ cb )(nwa , na , wa , ca ) ⊠ (nwb , nb , wb , cb ) = (nwa , na ⊗ nb , wa , ca ⊗ cb )(nwa , na , wa , ca ) ⊠′ (nwb , nb , wb , cb ) = (na ⊗ nwb , na ⊗ nb , ca ⊗ wb , ca ⊗ cb )p4 a = (a, a, a, a)
Proof. The theorem is proved by induction on the structure of two-dimensional
arrays. The base case is proved as follows.
scan (⊕, ⊗) |a|={ Definition of scan }|a|={ Definition of π1 , p4 , map and DH2 }(map π1 ◦ DH2 (⊞, ⊞′ , ⊠, ⊠′ ) ◦ map p4 ) |a|6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays

145

The inductive case for −◦ is proved as follows.
=
=

=

=

=

=

=

=

scan (⊕, ⊗) (x1 −◦ x2 ){ Definition of scan }scan (⊕, ⊗) x1 −◦ mapr (zipwith (⊕) (bottom (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ){ Hypothesis of induction }(map π1 ◦ DH2 (⊞, ⊞′ , ⊠, ⊠′ ) ◦ map p4 ) x1−
◦ mapr (zipwith (⊕) (bottom (scan (⊕, ⊗) x1 )))((map π1 ◦ DH2 (⊞, ⊞′ , ⊠, ⊠′ ) ◦ map p4 ) x2 ){ The property shown below, and function composition }map π1 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 ))−◦ zipwith (⊕) (map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )))(map π1 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ))){ Extracting the same terms using ‘where’ }map π1 x′1 −◦ zipwith (⊕) (map π3 x′1 ) (map π1 x′2 )where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )
π1 ((nwa , na , wa , ca ) ⊞′ (nwb , nb , wb , cb ))= wa ⊕ nwb = π3 (nwa , na , wa , ca ) ⊕ π1 (nwb , nb , wb , cb )map π1 x′1 −◦ map π1 (zipwith (⊞′ ) x′1 x′2 )where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )
π1 ((nwa , na , wa , ca ) ⊞ (nwb , nb , wb , cb ))= nwa = π1 (nwa , na , wa , ca )map π1 (zipwith (⊞) x′1 x′2 )−◦ map π1 (zipwith (⊞′ ) x′1 x′2 )where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Definition of map }map π1 (zipwith (⊞) x′1 x′2 −◦ zipwith (⊞′ ) x′1 x′2 )′where x1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Definition of DH2 and map }map π1 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 (x1 −◦ y1 )))

To complete the proof, we have to show the following property.

mapr (zipwith (⊕) (bottom (scan (⊕, ⊗) x)))= zipwith (⊕) (map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x)))
It is proved by three properties: (1) if all rows of z are the same, then the equation
zipwith (⊕) z y = mapr (zipwith (bottom z)) y
holds, (2) all rows of map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y) are the same, and (3) the
following equation holds.
bottom (scan (⊕, ⊗) x) = bottom (map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x)))

146

6. Implementation of Skeletons and Optimizations

The first one is obvious since each row of zipwith (⊕) z y is equal to the resultof application of zipwith (⊕) (bottom z) to the corresponding row of y.
The second one is proved by induction as follows. The base case is obvious. The
inductive case for −◦ is as follows.
map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (y1 −◦ y2 ))={ Definition of DH2 and map }map π3 (zipwith (⊞) y1′ y2′ )−◦ map π3 (zipwith (⊞′ ) y1′ y2′ )′where y1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y1
y2′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y2=
{ Definition of ⊞ and ⊞′ , promotion of map }zipwith (⊕) (map π3 y1′ ) (map π3 y2′ )−◦ zipwith (⊕) (map π3 y1′ ) (map π3 y2′ )where y1′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y1
y2′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y2−◦

By hypothesis of induction, all rows of the final equation are the same. The inductive
case for is as follows.

−
◦

−
◦

−
◦

map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (y1 y2 ))={ Definition of DH2 and map }map π3 (zipwith (⊠) y1′ y2′ ) map π3 (zipwith (⊠′ ) y1′ y2′ )where y1′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y1
y2′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y2=
{ Definition of ⊠ and ⊠′ , promotion of map }map π3 y1′ zipwith (⊗) (map π3 y1′ ) (map π3 y2′ )where y1′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y1
y2′ = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y2
By hypothesis of induction, all rows of the final equation are the same. Similarly,
we can prove that all columns of map π2 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y) are the same, andthat all elements of map π4 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) y) are the same.
The third property is proved by induction. The base case is obvious. The
inductive case for −◦ as follows.
bottom (scan (⊕, ⊗) (x1 −◦ x2 )){ Definition of scan and bottom }bottom (mapr (zipwith (⊕) (bottom (scan (⊕, ⊗) x1 ))) (scan (⊕, ⊗) x2 ))={ Promotion of bottom through mapr }mapr (zipwith (⊕) (bottom (scan (⊕, ⊗) x1 ))) (bottom (scan (⊕, ⊗) x2 ))={ Application of mapr f to a row vector is application of f }zipwith (⊕) (bottom (scan (⊕, ⊗) x1 )) (bottom (scan (⊕, ⊗) x2 )=

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays

=

=

=

=

{ Hypothesis of induction }zipwith (⊕) (bottom (map π3 x′1 )) (bottom (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Promotion of zipwith }bottom (zipwith (⊕) (map π3 x′1 ) (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Definition of ⊞′ , and promotion of zipwith }bottom (map π3 (zipwith (⊞′ ) x′1 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Definition of bottom }bottom (map π3 (zipwith (⊞) x′1 x′2 )−◦ map π3 (zipwith (⊞′ ) x′1 x′2 ))′′′where x1 = DH2 (⊞, ⊞ , ⊠, ⊠ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 ){ Definition of DH2 and map }bottom (map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 (x1 −◦ x2 )))))The inductive case foras follows.

−
◦

=

147

−
◦

bottom (scan (⊕, ⊗) (x1 x2 )){ Definition of scan, distribution of bottom }bottom (scan (⊕, ⊗) x1 )mapc (zipwith (⊕) (bottom (right (scan (⊕, ⊗) x1 )))) (bottom (scan (⊕, ⊗) x2 ))= { Hypothesis of induction, property of the fourth element }bottom (map π3 x′1 )mapc (zipwith (⊕) (bottom (right (map π4 x′1 )))) (bottom (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )= { bottom ◦ right = right ◦ bottom }bottom (map π3 x′1 )mapc (zipwith (⊕) (right (bottom (map π4 x′1 )))) (bottom (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )= { All elements of map π4 x′1 are the same }bottom (map π3 x′1 ) zipwith (⊕) (bottom (map π4 x′1 )) (bottom (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )= { Promotion of zipwith }bottom (map π3 x′1 ) bottom (zipwith (⊕) (map π4 x′1 ) (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )
−
◦

−
◦

−
◦

−
◦

−
◦

=

148

6. Implementation of Skeletons and Optimizations{ Distributivity of bottom over }bottom (map π3 x′1 zipwith (⊕) (map π4 x′1 ) (map π3 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )= { Definition of ⊠ and ⊠′ }bottom (map π3 (zipwith (⊠) x′1 x′2 zipwith (⊠′ ) x′1 x′2 ))where x′1 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x1 )x′2 = DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x2 )= { Definition of DH2 and map }bottom (map π3 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 (x1 x2 ))
−
◦

−
◦

−
◦

−
◦

=

The property of the fourth component is as follows.
bottom (right (scan (⊕, ⊗) x))= bottom (right (map π4 (DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x))))−◦This property is obvious because the bottom-right element of scan (⊕, ⊗) x is theresult of reduce (⊕, ⊗) x, and the fourth component of each element in the resultof DH2 (⊞, ⊞′ , ⊠, ⊠′ ) (map p4 x) is also the result of reduce (⊕, ⊗) x .
The inductive case for of the theorem is proved similarly.
Since the complexity of distributable homomorphism is O(T log(n)) and that of
the operators is T , this theorem says that scan can be implemented in parallel withthe complexity of O(T log(n)) . This implementation is used in global computationof the general implementation shown below. Figure 6.5 shows the implementationof scan by distributable homomorphism, where the number of steps is O(log(n)) .
Now, we will develop the general implementation, i.e., the segmented implementation of accumulate and scan, in which each processor has one subarray of the input
two-dimensional array rather than a single element. First, we will make the general implementation of accumulate. Then, we will instantiate it to make the general
implementation of scan.
Let’s consider to perform the main computation of accumulate independently on
processors. Each processor can perform its local computation on the subarray, when
it has the accumulation parameters necessary for its local computation. Thus, for thecomputation of accumulate, we first compute the accumulation parameters for eachprocessor by global computation, then perform the local computation independently
on each processor, and finally combine the local results by a global reduction.
The segmented parallel implementation of the accumulate is given by the following theorem. Note that we can use the simple case implementation of scan for
the global computation of the accumulation parameters for the local computations.
Theorem 6.2 (Segmented accumulate). The skeleton accumulate is executed seg-

149

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays(0,0,2,2)

0
P00

-1

-2

P01

2
P10

P02

-3
P03

1

0

P11

(0,0,0,0)

P12

P00

-1

(-1,-1,-1,-1)

P01(2,2,2,2)

p4

P13

P10

(-2,-2,-2,-2)

P02(1,1,1,1)

P11

P00

P03

(0,0,0,0)

P12

P01

;

0

P20

3

2

P21

6
P30

P22

1
P23

5

4

P31

(4,4,4,4)

P32

P20

3

(3,3,3,3)

P21(6,6,6,6)

P22(5,5,5,5)


P00



 )

(

;

0



0

P01

P30

P10

P02



P11

P31

P12

P32

(7,7,18,18)


P30

P31

 )

(

;

0


P32

P10

0

1

-1
P01

2
P10



P30

6
P20

12
P30

P10

P33

P13

0



0

(14,14,24,24)
P22(6,6,24,24)
P23



0





0

(24,24,24,24)(24,24,24,24)(20,24,20,24)
P31

0



(9,13,20,24)(12,24,12,24)
P30

P12

P21



0

(4,4,4,4)(-4,-4,24,24)(0,-4,24,24)



(6,18,12,24)

0

0

P11

P20



(-6,-6,24,24)
P03





(2,6,20,24)





0

P02

P32

P33

Step.5

-4
P13

14
P22

20
P31



P01

-6

0

9

0

0

(-2,-6,24,24)

P03

P12

P21

;

(-3,-3,4,4)(4,4,4,4)

Step.4

 )

0

P23

P32

(4,4,4,4)
P33





(2,6,12,24)



(0,0,4,4)

0

0(6,6,6,6)
P32(-1,3,20,24)


(

P13

P22

(20,20,20,20)
P31



0



0

(12,20,12,20)

P00(-6,-6,4,4)



(9,9,20,20)



0

-2

2

(8,8,8,8)(0,4,12,24)

P03

P12

P21



0

P31



(-5,-5,4,4)



0

P23



0

P30



(-2,-6,4,4)



(6,14,12,20)

P33

P02

P11

P11

P20

(10,10,10,10)

P02



0

Step.3

P00

P22



Step.2(2,2,20,20)



(3,3,10,10)(6,10,6,10)

P01



0

P23

0

(18,18,18,18)(3,3,3,3)(1,1,4,4)





0

P33

(-2,-6,4,4)



(2,2,12,20)



(2,6,6,10)
P22



(10,18,10,18)

0P13(-1,-1,20,20)

P00(-6,-6,-6,-6)



0

P21

(0,0,12,20)(-2,-6,-2,-6)

(4,12,10,18)
P20

0(-5,-5,-6,-6)
P03

0(2,2,2,2)(-2,-6,-2,-6)

(2,2,2,2)(4,4,4,4)

P21



0

(-4,-4,-4,-4)
P13(2,2,6,6)



Step.1(-1,-1,2,2)

P20

P23



0

(-2,-2,-2,-2)
P12(3,3,8,8)

(1,1,1,1)(10,10,10,10)

P33

Step.0(0,0,2,2)(2,2,2,2)



P03



(0,0,0,0)P11(4,4,10,10)

4

P02

0

(2,2,2,2)
P10(-3,-3,-4,-4)





0

 )

(

(-2,-2,-2,-2)(-1,-1,-1,-1)

P13(-1,-1,0,0)

(-3,-3,-3,-3)

6
P23

24
P32

24
P33

Step.6

Figure 6.5. Parallel Computation of scan (+, +) by Distributable Homomorphism
mentally in parallel as follows.

−
◦

−
◦

[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x (e, u, v)= reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip xs (zip3 es us vs)))where
xs = dist p q x
dus = dist 1 q u
dvs = dist p 1 v
z = (||e|| dus)−◦ (dvs fxs)
fxs = map (map f ) xs
us = dropc 1 (taker p (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z)))vs = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z)))es = taker p (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z)))
Proof. The theorem is proved by induction on division of the input two-dimensional

150

6. Implementation of Skeletons and Optimizations

array. The base case (p = q = 1) is proved as follows.

=

=
=
=

−
◦

−
◦

=

−
◦

RHS 1 1 x (e, u, v){ The right hand side }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip xs (zip3 es us vs)))where xs = dist 1 1 x
dus = dist 1 1 u
dvs = dist 1 1 v◦ (dvs fxs)z = (||e|| dus)−
qxs = map (map f ) xs
us = dropc 1 (taker 1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z)))vs = dropr 1 (takec 1 (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z)))es = taker 1 (takec 1 (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z))){ Definition of dist }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip |x| (zip3 es us vs)))◦ (|v| | map f x|)
where z = (||e|| |u|)−us = dropc 1 (taker 1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z)))vs = dropr 1 (takec 1 (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z)))es = taker 1 (takec 1 (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z))){ Definition of scan, takec , taker , dropc and dropr }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip |x| (zip3 es us vs)))where us = |u|, vs = |v|, es = |e|{ Definition of zip and map }reduce (⊕′ , ⊗′ ) |[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x (e, u, v)|{ Definition of reduce }[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x (e, u, v){ The left hand side }LHS x (e, u, v)−◦=

The inductive case for p = p1 + p2 (i.e. x = x1 −◦ x2 and v = v1 −◦ v2 ) is proved asfollows.
RHS (p1 + p2 ) q (x1 −◦ x2 ) (e, u, v1 −◦ v2 ){ The right hand side }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip xs (zip3 es us vs)))where xs = dist (p1 + p2 ) q (x1 −◦ x2 )
dus = dist 1 q u
dvs = dist (p1 + p2 ) 1 (v1 −◦ v2 )◦ (dvs fxs)z = (||e|| dus)−
qxs = map (map f ) xs
us = dropc 1 (taker (p1 + p2 ) (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z)))vs = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z)))es = taker (p1 + p2 ) (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z)))={ Definition of dist and , and introducing new variables }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip (xs 1 −◦ xs 2 ) (zip3 es us vs)))where xs 1 = dist p1 1 x1 , xs 2 = dist p2 1 x2
dus = dist 1 q u
dvs 1 = dist p1 1 v1 , dvs 2 = dist p2 1 v2
z1 = (||e|| dus)−◦ (dvs 1 fxs 1 ), z2′ = (dvs 2 fxs 2 )qxs 1 = map (map f ) xs 1 , qxs 2 = map (map f ) xs 2
us = dropc 1 (taker (p1 + p2 ) (scan (zipwith (⊕), ≫) (map (reducec (⊕)) (z1−◦z2′ ))))′
vs = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) (z1 −◦ z2 ))))
es = taker (p1 + p2 ) (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) (z1 −◦ z2′ ))))
−
◦

−
◦

−
◦

−
◦

−
◦

=

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

{ Properties shown below }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e)
(zip (xs 1 −◦ xs 2 ) (zip3 (es 1 −◦ es 2 ) (us 1 −◦ us 2 ) (vs 1 −◦ vs 2 ))))
where
e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map f x1 ))xs 1 = dist p1 1 x1 , xs 2 = dist p2 1 x2
dus = dist 1 q u
dus 2 = dist 1 q u2
dvs 1 = dist p1 1 v1 , dvs 2 = dist p2 1 v2
z1 = (||e|| dus)−◦ (dvs 1 fxs 1 ), z2 = (||e2 || dus 2 )−◦ (dvs 2 fxs 2 )qxs 1 = map (map f ) xs 1 , qxs 2 = map (map f ) xs 2
us 1 = dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))vs 1 = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z1 )))es 1 = taker p1 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z1 )))us 2 = dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2 )))vs 2 = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z2 )))es 2 = taker p2 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z2 )))={ Definition of reduce, map, and zip }reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip xs 1 (zip3 es 1 us 1 vs 1 )))⊕ reduce (⊕′ , ⊗′ ) (map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) (zip xs 2 (zip3 es 2 us 2 vs 2 )))wheree2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map f x1 ))xs 1 = dist p1 1 x1 , xs 2 = dist p2 1 x2
dus = dist 1 q u, dus 2 = dist 1 q u2
dvs 1 = dist p1 1 v1 , dvs 2 = dist p2 1 v2
◦ (dvs 1 fxs 1 ), z2 = (||e2 || dus 2 )−◦ (dvs 2 fxs 2 )z1 = (||e|| dus)−
qxs 1 = map (map f ) xs 1 , qxs 2 = map (map f ) xs 2
us 1 = dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))vs 1 = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z1 )))es 1 = taker p1 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z1 )))us 2 = dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2 )))vs 2 = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z2 )))es 2 = taker p2 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z2 )))={ The right hand side }RHS p1 q x1 (e, u, v1 ) ⊕ RHS p2 q x2 (e, u, v2 )where
e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map f x1 ))={ Hypothesis of induction }[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x1 (e, u, v1 ) ⊕ [[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x2 (e2 , u2 , v2 )where
e2 = e ⊕ reduce (⊕, ⊗) v1
u2 = zipwith (⊕) u (reducec (⊕) (map f x1 ))={ Definition of accumulate }[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] (x1 −◦ x2 ) (e, u, v1 −◦ v2 )={ The left hand side }LHS (x1 −◦ x2 ) (e, u, v1 −◦ v2 )
−
◦

=

151

152

6. Implementation of Skeletons and Optimizations

To complete the above proof, we show the following properties.
dropc 1 (taker (p1 + p2 ) (scan (zipwith (⊕), ≫) (map (reducec (⊕)) (z1 −◦ z2′ ))))= dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))−◦dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2 )))dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) (z1 −◦ z2′ ))))= dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z1 )))−◦dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z2 )))taker (p1 + p2 ) (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) (z1 −◦ z2′ ))))
= taker p1 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z1 )))−◦taker p2 (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z2 )))
−
◦

−
◦

−
◦

where
z1 = (||e|| dus)−◦ (dvs 1 fxs 1 ), z2 = (||e2 || dus 2 )−◦ z2′e2 = e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))These properties are proved using the following property (see the last part of proofof Theorem 3.1 .)bottom (scan (⊕, ⊗) y) = scan (⊕, ⊗) (reducec (⊕) y)
The first property is proved as follows.

−◦

dropc 1 (taker (p1 + p2 ) (scan (zipwith (⊕), ≫) (map (reducec (⊕)) (z1 −◦ z2′ ))))=
{ Definition of scan }dropc 1 (taker (p1 + p2 ) (s1 −◦ mapr (zipwith (zipwith (⊕)) (bottom s1 )) s2 ))where s1 = scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )s2 = scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2′ )={ Definition of taker , dropc and z1 }dropc 1 (taker p1 s1 )−◦dropc 1 (taker p2 (bottom s1 −◦ mapr (zipwith (zipwith (⊕)) (bottom s1 )) s2 ))where s1 = scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )s2 = scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2′ )={ The above property, and bottom ◦ bottom = bottom }dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))−◦dropc 1 (taker p2 (scan (zipwith (⊕), ≫) r−
◦ mapr (zipwith (zipwith (⊕)) (bottom (scan (zipwith (⊕), ≫) r))) s2 ))where r = reducec (zipwith (⊕)) (map (reducec (⊕)) z1 )s2 = scan (zipwith (⊕), ≫) (map (reducec (⊕)) z2′ )={ Definition of scan }dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))−◦dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (r −◦ map (reducec (⊕)) z2′ )))where r = reducec (zipwith (⊕)) (map (reducec (⊕)) z1 )={ Calculation of r shown below }dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))−◦dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (r −◦ map (reducec (⊕)) z2′ )))wherer = map (reducec (⊕)) (||e2 || dus 2 )e2 = e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays{ Introducing the variable z2 }dropc 1 (taker p1 (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z1 )))−◦dropc 1 (taker p2 (scan (zipwith (⊕), ≫) (r −◦ map (reducec (⊕)) z2 )))
where
◦ z2′
z2 = (||e2 || dus 2 )−e2 = e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))
−
◦

=

153

The calculation of r is as follows.
reducec (zipwith (⊕)) (map (reducec (⊕)) z1 ){ Definition of z1 }reducec (zipwith (⊕)) (map (reducec (⊕)) ((||e|| dus)−◦ (dvs 1 fxs 1 )))={ Definition of reducec and map }(zipwith (zipwith (⊕)) ||e|| (reducec (zipwith (⊕)) (map (reducec (⊕)) dvs 1 )))(zipwith (zipwith (⊕)) (map (reducec (⊕)) dus)
(reducec (zipwith (⊕)) (map (reducec (⊕)) 
fxs 1 )))

If width x = 1, then reducec (⊕) x = | reduce (⊕, ⊗) x|,=
reduce (zipwith (⊕, ⊗)) ◦ map | · | = | · | ◦ reduce (⊕, ⊗)(zipwith (zipwith (⊕)) ||e|| || reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )||)
(zipwith (zipwith (⊕)) (map (reducec (⊕)) dus)
(reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 )))={ If the height of each element of x is one, then map (reducec (⊕)) x = x }(zipwith (zipwith (⊕)) ||e|| || reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )||)map (reducec (⊕)) (zipwith (zipwith (⊕)) dus(reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 )))={ Definition of zipwith, introducing the variable dus 2 }||e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )|| map (reducec (⊕)) dus 2
where
dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))={ Definition of map and reducec , introducing the variable e2 }map (reducec (⊕)) (||e2 || dus 2 )where
e2 = e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))
−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

Now, the proof of the first property is completed.
The second property is proved similarly. Assuming that height y1 = 1, the key
calculation is as follows.
dropr 1 (scan (≫, zipwith (⊗)) (y1 −◦ y2 )){ Definition of scan }dropr 1 (scan (≫, zipwith (⊗)) y1−◦ mapr (zipwith (≫) (scan (≫, zipwith (⊗)) y1 )) (scan (≫, zipwith (⊗)) y2 ))={ Drop the top row }mapr (zipwith (≫) (scan (≫, zipwith (⊗)) y1 )) (scan (≫, zipwith (⊗)) y2 ))={ zipwith (≫) x y = y }scan (≫, zipwith (⊗)) y2=

154

6. Implementation of Skeletons and Optimizations
The third property is proved similarly. The key calculation is as follows.
reducec (⊕) (map (reduce (⊕, ⊗)) z1 ){ Definition of z1 }◦ (dvs 1 fxs 1 )))reducec (⊕) (map (reduce (⊕, ⊗)) ((||e|| dus)−={ Definition of reducec and map }(zipwith (⊕) |e| (reducec (⊕) (map (reduce (⊕, ⊗)) dvs 1 ))))
(zipwith (⊕) (map (reduce (⊕, ⊗)) dus) (reducec (⊕) (map (reduce (⊕, ⊗)) fxs 1 )))={ If width x = 1, then reducec (⊕) x = | reduce (⊕, ⊗) x| }(zipwith (⊕) |e| | reduce (⊕, ⊗) (map (reducec (⊕)) dvs 1 )|)(zipwith (⊕) (map (reduce (⊕, ⊗)) dus) (reducec (⊕) (map (reduce (⊕, ⊗)) fxs 1 )))={ Definition of zipwith, moving the application of ⊗ to the front }|e ⊕ reduce (⊕, ⊗) (map (reducec (⊕)) dvs 1 )|map (reduce (⊕, ⊗)) (zipwith (zipwith(⊕)) dus(reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 )))={ Introducing e2 and dus 2 , definition of map and reduce }map (reduce (⊕, ⊗)) (||e2 || dus 2 )where
e2 = e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 )dus 2 = zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 ))
−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

Finally, the following calculation completes the proof.
zipwith (zipwith (⊕)) dus (reducec (zipwith (⊕)) (map (reducec (⊕)) fxs 1 )){ Definition of dus and fxs 1 }zipwith (zipwith (⊕)) (dist 1 q u)
(reducec (zipwith (⊕)) (map (reducec (⊕)) (map (map f ) (dist p q x))))={ Moving dist to the front }dist 1 q (zipwith (⊕) u (reducec (⊕) (map f x)))=e ⊕ reduce (⊕, ⊗) (map (reduce (⊕, ⊗)) dvs 1 ){ Definition of dvs 1 and (reverse) promotion of reduce (⊕, ⊗) }e ⊕ reduce (⊕, ⊗) (reduce (−◦, ) (dist p q v1 ))={ gather = reduce (−◦, ) and gather ◦ dist p q = id }e ⊕ reduce (⊕, ⊗) v1
−
◦

−
◦

=

The inductive case for q = q1 + q2 is proved similarly.
The computation of us, vs and es is done using the previous implementationof scanp by distributable homomorphism, in which the complexity of the operators
is O( n/(pq)) p
for an n × n array and pq processors. Thus, its computational
complexity is O( n/(pq) log(pq)) . The computation of reduce (⊕′ , ⊗′ ) is O(log(pq))
using tree-like computation, and that of map (λ(x, e).[[(⊕′ , ⊗′ , f ′ ), (⊕, ⊗, f )]] x e) isO(n2 /(pq)) since it is performed
independently on each processor. Thus, totalpcomplexity is O(n2 /P + n2 /P log(P )) for an n × n array and P = pq processors.
Note that we can compute us, vs, and es simultaneously with a single scan by
tupling them, since they have the same structure of computation.
How, we can obtain the segmented implementation of scan using the theorem.
Corollary 6.3 (Segmented scan). The skeleton scan is executed segmentally in6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays

155

parallel as follows.

−
◦

−
◦

−
◦

−
◦

−
◦

scan (⊕, ⊗)= gather (map (λ(x, e).[[(−◦, , λx e.e), (⊕, ⊗, f )]] x e) (zip xs (zip3 es us vs)))where
xs = dist p q x
dus = dist 1 q (|ı| · · · |ı|)dvs = dist p 1 (|ı|−◦ · · ·−
◦ |ı|)z = (||e|| dus)−◦ (dvs fxs)
fxs = map (map f ) xs
us = dropc 1 (taker p (scan (zipwith (⊕), ≫) (map (reducec (⊕)) z)))vs = dropr 1 (takec q (scan (≫, zipwith (⊗)) (map (reducer (⊗)) z)))es = taker p (takec q (scan (⊕, ⊗) (map (reduce (⊕, ⊗)) z)))where ı ⊕ a = a ⊕ ı = a, ı ⊗ a = a ⊗ ı = a−◦p
The computational complexity of scan is O(n2 /P + n2 /P log(P )) for an n × narray and P processors, which is equal to that argued in previous section. Note that[[(−◦, , λx e.e), (⊕, ⊗, f )]] is scan except that it takes the initial values (accumulation
parameters.)

6.2.3

Refined Implementation for Nested Use of Skeletons

The ordinary implementation for tree-like computation of the skeleton reduce uses
only one processor to each operator. Precisely saying, one processor receives thevalue of the other processor and applies the operator to the pair of its value and thereceive value by only itself, while the other processor becomes idle after sending its
value.
This becomes a problem when the complexity is not small, although it is not a
problem when the computational complexity of the operator is small (e.g., O(1) orO(log(n)) for the input of size n). For example, the program mrs for the maximum
rectangle sum problem in Section 4.3.3 has a reduce with operators of which complexity is O(n3 ) for the input matrix of n × n. Its parallel computational complexity
for the ordinary implementation using P processors is as follows:
T (n × n, P ) = n3 + T (n × n/2, P/2) = O(n3 ) .
Since the complexity of the last operator is big and dominant, the parallel complexity
does not depend on the number of processors P . Thus, this program is not efficiently
executed in parallel even if the number of processors becomes large. Generally, thisproblem arises when the complexity of the operator is equal or bigger than O(n).
This situation occurs when the operator contains skeletons, i.e., when the program
contains nested uses of skeletons.
The main cause of this problem is that the implementation uses only one processor for each operator in the tree-like computation; some processors remain idle

156

6. Implementation of Skeletons and Optimizations

during the computation. If we can use these idle processors for the computation of
reduction operators, we may improve the efficiency.
We will use those idle processors to solve this problem. In the refined implementation, a set of available processors is attached to each operator of the reduce
skeleton so that these processors are used for the parallel execution of the skeletonsof the operator. Furthermore, since map is often used to describe explicit parallelization of independent computations, we also give refined implementation of map
having a set of processors.
In the following, f P means performing the computation of f to a distributed
matrix using the set of processors P in parallel, and f means to distribute thePargument matrix and perform the computation f using P in parallel.
We formalize the computation of skeletons with the attached set of processors.
An outermost skeleton marked by ·P distributes the argument matrix among the setof processors P (|P | ≥ p × q) and does local computation in parallel by using map
marked by ·P . The reduce performs global tree-like computation marked by ·P .
map fPreduce (⊕, ⊗)P= gather ◦map (map f )P ◦ dist p q= reduce (⊕, ⊗)P ◦ map (reduce (⊕, ⊗))P ◦ dist p q−◦

The map marked by ·P divides the set of available processors P into two sets of P1
and P2 in the cases of −◦ and , and performs the computation of f in parallel on P
in the base case. The division of P is done so that the ratio of the sizes of P1 andP2 is the same of that of x and y.

−
◦

−
◦

map f P |a|= f aP◦ y) = map f P x −◦ map f P y
map f P (x−12
map f P y
map f P (x y) = map f P x
1

2

−
◦

The reduce marked by ·P for the global computation divides the set of available
processors P into two sets of P1 and P2 in the cases of −◦ and , and performs
the computation of each operator in parallel on P that is the set of all available
processors for the operator. The division of P is done so that the ratio of the sizesof P1 and P2 is the same of that of x and y.

−◦

reduce (⊕, ⊗)P |a|= areduce (⊕, ⊗)P (x−◦ y) = reduce (⊕, ⊗)P x ⊕ reduce (⊕, ⊗)P yP
1
2
reduce (⊕, ⊗)P (x y) = reduce (⊕, ⊗)P x ⊗ reduce (⊕, ⊗)P y
1

P

2

This refined implementation enables parallel execution of the last operator in thetree-like computation. It can reduce the dominant complexity, and thus, improve
the parallel complexity of the program.
Using this refined implementation, computational complexity of mrs using P
processors is as follows:
T (n × n, P ) = n3 /P + log(P ) + T (n × n/2, P/2) = O(n3 /P + log(P )) .

6.2. Implementation of Parallel Skeletons on Two-dimensional Arrays157

The parallel complexity of the last operator is inversely proportional to P because
the computation of the operator is performed in parallel among P processors. Thus,
this program is efficiently executed in parallel as the number of processors becomeslarge.

6.2.4

Experiment Results

We implemented the parallel skeletons as a library 1 with C++ and MPI, and did
our experiments on a small-scale PC cluster of sixteen uniform PCs connected with
Gigabit Ethernet. Each PC has a CPU of Pentium4 3.0GHz (Hyper ThreadingON) and 1GB memory, with Linux 2.6.8 for the OS, gcc 3.4 for the compiler, andmpich 1.2.6 for the MPI.
Figures 6.6 through 6.8 show speedups of the following parallel programs described with the parallel skeletons.
Frobenius Norm
fnorm = reduce (+, +) ◦ map square ,Matrix Multiplication
mm (composition of skeletons; see Section 3.2.2).
Maximum Rectangle Sum
mrs (derived program written by skeletons; see Section 4.3.3).
Image Filter
ifilter = map f ◦ scanr (⊕′r , ⊗′r ) ◦ map fr ◦ scan(⊕′f , ⊗′f ) ◦ map ff(optimized program written by composition of skeletons); see Section 5.3.2.
The inputs are a 4000 × 4000 matrix for fnorm, , 1000 × 1000 matrices for mm andmrs, and a 2000 × 2000 matrix for ifilter . The computational times of the above
programs on one processor are 0.21s, 2.06s, 14.1s, and 116.8s respectively.
The result shows programs described with skeletons can be executed efficiently
in parallel, and proves the success of our framework. Both of the simple program(fnorm) and the more complicated program (ifilter ) achieve linear speedups by theparallel implementation of the skeletons. The speedup of matrix multiplication issuper-linear. This can happen in large matrix operations where the matrix on asingle processor is large with respect to the cache size. It is reasonable that thesuper-linear speedup is achieved here. Two lines in Figure 6.8 are the results of mrs
using the simple implementation (labeled ‘mrs’) and the refined implementation
for nested use of skeletons (labeled ‘mrs(refined)’), respectively. The refined one
gives us a good speedup nearly proportional to the number of processors for 16
processors, while the speedup of the simple one is reaching the limit soon. This
result indicates that even if a program written by the skeletons contains nested useof skeletons, we can execute it efficiently in parallel when we successfully execute
1This implementation is included in the skeleton library ‘SkeTo’. It is available at the web page
http://www.ipl.t.u-tokyo.ac.jp/sketo/158

6. Implementation of Skeletons and Optimizations

the inner skeletons in parallel. However, even for the refined implementation of mrs,
the speedup is expected to reach the limit earlier than other examples, because it
uses heavy operators and needs to transfer larger data structures among processors.
Finally, we list a part of the C++ code of mm written with the skeleton libraryto give a concrete impression of the conciseness our library provides. Figure 6.10
shows the C++ code of the function mm that implements mm. The function takes
three distributed matrices (i.e., two-dimensional arrays): Z2 to store the result,and X2 and Y2 for the input. Two functions all rows and all cols implements
the corresponding functions in mm with provided skeletons. The results of thesefunctions are stored in two arrays A2 and B2. These arrays are then supplied to theprovided skeleton zipwith, which is an implementation of skeleton zipwith. Here,
users do not need to be conscious of its parallel implementation; what they needto do is simply to supply two distributed matrices and the function object to be
applied to elements.

6.3

Implementation of Fusion Optimizations

In this section, we will briefly report our simple implementations of fusion optimizations: the domain-independent fusion optimization in Section 4.1.3, and thedomain-specific fusion optimization in Section 5.1. We will also report some experiment results to show the effect of optimizations.

6.3.1

A Simple System for Domain-Independent Fusion Optimization

SkeTo [MIEH06] library is equipped with the domain-independent fusion optimization shown in Section 4.1.3. We will briefly review its mechanism.
The optimization mechanism consists of three components: the user-interface
database, the transformation engine, and the implementation database. Taking a
C++ program with skeletons, the transformation system first converts the skeletons
into their structured forms (i.e., accumulate and buildJ in Section 4.1.3) by applying
rules given as meta-programs in the user-interface database. Then, the transformation engine manipulates and fuses the structured forms with the shortcut fusion
rules (Theorem 4.4 and its variants). Finally, the system generates optimized skeleton compositions from the fused structured forms by the rules in the implementation
database.
The optimization system is implemented in OpenC++, a meta language formanipulating C++ programs.
For example, the following two lines of code to apply two functions f and g to
the list x
y = x->map(f);z = y->map(g);159

6.3. Implementation of Fusion Optimizations

18
fnorm
linear16
14
speedup

12
10
8
6
4
2
0
1

2

4

8
processors

16

Figure 6.6. Speedup of F-norm

30
mm
linear25

speedup

20
15
10
5
0
1

2

4

8
processors

Figure 6.7. Speedup of matrix multiplication

16

160

6. Implementation of Skeletons and Optimizations

18
mrs(refined)mrs
linear16
14
speedup

12
10
8
6
4
2
0
1

4

9
processors

16

Figure 6.8. Speedup of maximum rectangle sum

18
imagefilter
linear16
14
speedup

12
10
8
6
4
2
0
1

4

8
processors

Figure 6.9. Speedup of Image Filter

166.3. Implementation of Fusion Optimizations

161

template < lass C, lass A, lass B>
void mm(dist_matrix<C> &Z2, onst dist_matrix<A> &X2, onst dist_matrix<B> &Y2){
dist_matrix < matrix < int > > *A2;
dist_matrix < matrix < int > > *B2;
A2 = all_rows2(X2);B2 = all_ ols2(Y2);m_skeletons::zipwith(Iprod<C>(), *A2, *B2, Z2);
delete B2;
delete A2;}

Figure 6.10. C++ code of matrix multiplication described with parallel skeletons. dist matrix is the type of distributed tow-dimensional arrays. Two functions all rows and all cols are also implemented with skeletons. The function
m skeleton::zipwith is the implementation of zipwith skeleton.
is converted by OpenC++ into the following parse tree , which is a nested list of
tokens.
[[sqs = [[subs -> map] ( [sq] )]] ;][[sqs = [[subs -> map] ( [sq] )]] ;]Then, those parse trees are transformed into structured forms by rules given as
meta-programs in the user-interface database as follows.
[‘buildJ‘ y x [[var c] [var s [f]] [var e]] ;][‘buildJ‘ z y [[var c] [var s [g]] [var e]] ;]Here, the two maps are structured by the buildJ . The transformation engine thentransform the structured forms into the following one structured form, by applying
the fusion rules.
[‘buildJ‘ z y [[var c] [var s [g] [f]] [var e]] ;]Finally, the system transform the structured form back into the following invocationof the skeleton.
y = x->map(compose(g, f));Here, compose is a function to compose two functions. The system has a map totransform structured forms into skeletons.

6.3.2

A Simple System for Domain-Specific Fusion Optimization

The system reads a skeleton program written with skeleton library SkeTo [MIEH06],
and generates an optimized C++ code of the program. The optimized code is writtenwith direct use of MPI or skeletons in SkeTo library.
The optimization flow of the system is as follow.

162

6. Implementation of Skeletons and Optimizations

Phase. 1 Parsing the input program to extract skeleton compositions.
Phase. 2 Building normal forms from the extracted skeleton compositions by domainspecific fusions.
Phase. 3 Generating optimized codes of efficient implementations to replace theskeleton compositions in the input program.
We will explain the flow with the domain-specific fusion developed in Section 5.2.
In the first phase, the system reads C++ program written with skeletons provided by skeleton library SkeTo. Figure 6.11 shows an example C++ program that
implements the running example next in Section 5.2 as the C++ function nextZW,
which uses zipwith instead of the composition of map and zip.
Then, the system extracts skeleton compositions of specific patterns from theC++ program. For the program in Figure 6.11, it extracts an instance of Program,i.e., the following composition.
map add (zip (map add (zip (map c 2x (shift≫ b0 (shift≫ b1 u))) (map c 1x (shift≫ b1 u))))(map add (zip (map c0x u) (map c1x (shift≪ b2 u)))))
When there are more than one patterns of compositions (i.e., domains of computations), the system tests those patterns one by one. The unit of extracting skeleton
compositions is a function declaration in the input program.
In the second phase, the system transforms the extracted compositions into normal forms by using fusions specific to the compositions patterns. For example, theabove extracted composition is transformed into the following normal form by thefusion rules defined in Section 5.2 (the fusion rules can work well as Haskell functions
to carry out the transformation).
−→−→[[ [ add (add (c 2x(b0), c 1x(b1)), add (c0x(u[0]), c1x(u[1]))),
−
→
−
→
−
→
add (add (c 2x(b1), c 1x(u[0])), add (c0x(u[1]), c1x(u[2])))],add (add (c 2x(u≪2 ), c 1x(u≪1 )), add (c0x(u), c1x(u≫1 ))),
←
−
←
−
←
−
[add (add (c 2x(u[2]), c 1x(u[1])), add (c0x(u[0]), c1x(b2)))] ]]The the last phase, the system generates C++ code of efficient implementationsof built normal forms, to replace the skeleton compositions with these implementations. The replacement code is written with direct use of MPI or skeletons provided
by SkeTo library. Figure 6.12 shows the generated code to replace the function
nextZW of Figure 6.11, which implements the efficient implementation of the above
normal form discussed in Section 5.2.
Finally, the system outputs the input C++ program of which skeleton compositions are replaced with the generated code.

6.3. Implementation of Fusion Optimizations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

163

// ...
// definitions of parameter functions
struct times t : public skeleton::unary function<double,double>{const double c;times t(double c) : c(c) {}double operator()(double r) const{
return c * r;}
} c 1x(c 1), c0x(c0);struct ADD : public skeleton::binary function<double,double,double>{double operator()(double x, double y) const {return x+y;}
}add;//start of skeleton program
dist list<double> *nextZW(dist list<double> *u) {dist list<double> *sl1 = list skeletons::shiftl(b2,u);dist list<double> *sr1 = list skeletons::shiftr(b1,u);dist list<double> *sr2 = list skeletons::shiftr(b0,sr1);
dist list<double> *msl1 = list skeletons::map(c1x,sl1);
dist list<double> *m0 = list skeletons::map(c0x,u);
dist list<double> *msr1 = list skeletons::map(c 1x,sr1);
dist list<double> *msr2 = list skeletons::map(c 2x,sr2);
dist list<double> *sz1 = list skeletons::zipwith(add, msr2, msr1);
dist list<double> *sz2 = list skeletons::zipwith(add, m0, msl1);
dist list<double> *res = list skeletons::zipwith(add, sz1, sz2);
delete sz2;
delete sz1;
delete msr2;
delete msr1;
delete m0;
delete msl1;
delete sr2;
delete sr1;
delete sl1;
return res;}//end of skeleton program// the main function
int SketoMain(int argc, char **argv) {// ...
for(int t=0;t<count;t++){tmp = nextZW(ary);
delete ary;
ary = tmp;}// ...
return 0;}

Figure 6.11. The input source code written with skeletons in SkeTo library (someparts are omitted). The function nextZW corresponds to the running example next
in Section 5.2.

164

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

6. Implementation of Skeletons and Optimizations

dist list<double> *nextZW(dist list<double> *u) {dist list<double> *res = new dist list<double>(n);enum { TAG1, TAG2 };const int L = 1;
const int R = 2;
const int rank = skeleton::rank;
const int procs = skeleton::procs;
const int localn = u->get local size(rank);
double *uData = &(u->at(0));
double *resData = &(res->at(0));
double *sbufl = new double[L];double *rbufl = new double[R];double *sbufr = new double[R];double *rbufr = new double[L];MPI Request reqs1, reqr1, reqs2, reqr2;
if(rank!=0){
for(int i = 0; i < L; i++) { sbufl[i] = uData[i]; }MPI Isend(sbufl,L,MPI DOUBLE,rank-1,TAG2,MPI COMM WORLD,&reqs1);
MPI Irecv(rbufl,R,MPI DOUBLE,rank-1,TAG1,MPI COMM WORLD,&reqr1);}
if(rank!=procs-1){
for(int i = 0; i < R; i++) { sbufr[i] = uData[localn-R+i]; }MPI Isend(sbufr,R,MPI DOUBLE,rank+1,TAG1,MPI COMM WORLD,&reqs2);
MPI Irecv(rbufr,L,MPI DOUBLE,rank+1,TAG2,MPI COMM WORLD,&reqr2);}
if(rank==0) {resData[0] = add(add(c 2x(b0), c 1x(b1)), add(c0x(uData[0]), c1x(uData[1])));
resData[1] = add(add(c 2x(b1), c 1x(uData[0])), add(c0x(uData[1]), c1x(uData[2])));}
if(rank==procs-1) {resData[localn-1-0] = add(add(c 2x(uData[localn-1-2]), c 1x(uData[localn-1-1])), add(c0x(uData[localn-1-0]), c1x(b2)));
}

33
34
35
36

for(int i = R; i < localn-L; i++) {resData[i] = add(add(c 2x(uData[i-2]), c 1x(uData[i-1])), add(c0x(uData[i+0]), c1x(
uData[i+1])));}
if(rank!=0){ MPI Wait(&reqr1,MPI STATUS IGNORE); }if(rank!=procs-1){ MPI Wait(&reqr2,MPI STATUS IGNORE); }if(rank!=0) {resData[0] = add(add(c 2x(rbufl[0]), c 1x(rbufl[1])), add(c0x(uData[0]), c1x(uData[1])));
resData[1] = add(add(c 2x(rbufl[1]), c 1x(uData[0])), add(c0x(uData[1]), c1x(uData[2])));}if(rank!=procs-1) {resData[localn-1-0] = add(add(c 2x(uData[localn-1-2]), c 1x(uData[localn-1-1])), add(c0x(uData[localn-1-0]), c1x(rbufr[0])));}
delete [] sbufl; delete [] rbufl; delete [] sbufr; delete [] rbufr;
return res;

37
38
39
40
41
42
43
44
45
46
47
48
49

}

Figure 6.12. The output optimized code to replace the function nextZW of Figure 6.11.

6.3. Implementation of Fusion Optimizations

6.3.3

165

Experiment Results

We will show experiment results of the domain-specific fusion developed in Section 5.2 to evaluate its effectiveness.
We measured running times of naively-composed skeleton programs and theiroptimized programs for the examples next and solveTS in Section 5.2 as well as thefollowing simpler example upwind .
upwind u = zipwith (+) (shift≫ 0 (map (c1 ×) u)) (map (c0 ×) u)
Here, each element of the output is computed from its corresponding element in
the input u and its left element. We used a PC cluster where each of the nodes
connected with Gigabit Ethernet has a CPU of Intel R Xeon R 2.80GHz and 2GB
memory, with Linux 2.4.21, GCC 4.1.1, and mpich 1.2.7.
Result of upwind
Table 6.1 lists measured running times and speedups, and Figure 6.13 shows the
speedups. Running time is of applying the function 10 times to an input list of
40,000,000 elements. A speedup is a ratio of running time of a parallel program torunning time of a sequential program of a single for-loop.
The program upwind f optimized by the domain-independent fusion achieves
two times faster running time than the original skeleton program upwind. The
program upwind dsf optimized with the domain-specific optimization achieves ten
times faster running time than the original skeleton program, and the same running
time as a sequential program on one processor. These improvements were gained byelimination of redundant intermediate data and communications.
Precisely saying, the domain-independent optimization reduces the number oftraversals on lists from four in the naive program to two in the optimized program.
Actually, the optimized program contains only one skeleton accumulate. However,the skeleton essentially traverses the list twice. Therefore, the reduction of thenumber of traversals is limited to two but not one. Note that the computation(multiplications of constants and additions) are far cheaper than memory access inthis example program.
On the other hand, the domain-specific optimization reduces the number oftraversals on lists from four to one, because it dispatches implementation specific tothe computation pattern. Therefore, the domain-specific optimization achieves fourtimes faster running time.
Also, the optimized program achieves good speedups against the number of processors. The drop of speedup is expected to be suppressed when the computation iscarried out on larger input.
These results guarantee effectiveness of the proposed optimization.

166

6. Implementation of Skeletons and Optimizations

speedup w.r.t. sequential program

Table 6.1. Running times and speedups of upwind (naive skeleton program), upwind f (optimized by fusion), and upwind dsf (optimized by DS fusion) against
the number of processors. A speedup is one with respect to a sequential program.
#processors
1
2
4
8
16
32
64
upwindtime (s) 32.97 16.14 8.13 3.95 1.97 1.05 0.73
speedup 0.26 0.53 1.05 2.15 4.31 8.14 11.66
upwind f time (s) 10.13 5.07 2.60 1.46 0.65 0.47 0.35
speedup 0.84 1.68 3.27 5.85 13.00 17.94 24.03
upwind dsf time (s)8.51 4.14 2.08 1.04 0.51 0.28 0.17
speedup 1.00 2.06 4.09 8.21 16.57 30.04 49.98

60
50

upwindupwind_f
upwind_dsf
linear

40
30
20
10
0
14 8

16

24 32
#processors

48

64

Figure 6.13. Speedups of upwind (naive skeleton program), upwind f (optimized
by fusion), and upwind dsf (optimized by DS fusion) with respect to the numberof processors.

6.3. Implementation of Fusion Optimizations

167

r[0] = c_2*b0 + c_1*b1 + c0*u[0] + c1*u[1];r[1] = c_2*b1 + c_1*u[0] + c0*u[1] + c1*u[2];for(int i = 2; i < n-1; i++) {r[i] = c_2*u[i-2] + c_1*u[i-1] + c0*u[i] + c1*u[i+1];}r[n-1] = c_2*u[n-3] + c_1*u[n-2] + c0*u[n-1] + c1*b2;
Figure 6.15. A sequential program of next
Result of next
Table 6.2 lists measured running times and speedups, and Figure 6.14 shows the
speedups. Running time is of applying the function 100 times to an input list of
10,000,000 elements. A speedup is a ratio of running time of a parallel program torunning time of a sequential program (shown in Figure 6.15). It is worth notingthat the domain-independent fusion does not work for this example, and thus wehave no result for the domain-independent fusion here.
The optimized program next dsf achieves ten times faster running time than theoriginal skeleton program next, and the same running time as a sequential programon one processor. This improvement was gained by elimination of redundant intermediate data and communications. Precisely saying, the optimization reduces thenumber of traversals on lists from ten in the naive program (see Figure 6.11) to one
in the optimized program. Therefore, the optimized program runs ten times fasterthan the naive program. Note that the computation (multiplications of constants
and additions) are far cheaper than memory access in this example program.
Also, the optimized program achieves good speedups against the number of processors. The speedup drops earlier than the previous example upwind, because
the optimized implementation of next performs communication twice, and thus its
overhead appears earlier, while the optimized program of upwind performs communication once. The drop of speedup can be suppressed when the computation iscarried out on larger input.
These results guarantee effectiveness of the proposed optimization.
Result of solveTS
Table 6.3 lists measured running times and speedups, and Figure 6.16 shows the
speedups. Running time is of applying the function 10 times to an input list of
1,000,000 elements. A speedup is a ratio of running time of a parallel program torunning time of a sequential program
The optimized program solveTS dsf achieves about 20% faster running timethan the original skeleton program solveTS. This improvement was gained by elimination of redundant intermediate data and communications, although its effectiveness is small because both programs have the same heavy computation of scan.
Also, the optimized program achieves good speedups against the number of pro-

168

6. Implementation of Skeletons and Optimizations

speedup w.r.t. sequential program

Table 6.2. Running times and speedups of next (naive skeleton program) andnext dsf (optimized by DS fusion) against the number of processors. A speedup is
one with respect to a sequential program.
#processors
1
2
4
8
16
32
64
next
time (s) 210.25 100.84 48.12 24.41 13.31 6.52 3.50
speedup
0.09
0.20 0.41 0.81 1.49 3.04 5.67
next dsf time (s)19.86
9.64 4.93 2.44 1.26 0.70 0.47
speedup
1.00
2.06 4.03 8.14 15.79 28.26 42.22

60
50

next
next_dfs
linear

40
30
20
10
0
14 8

16

24

32

48

64

#processors

Figure 6.14. Speedups of next (naive skeleton program) and next dsf (optimized
by DS fusion) with respect to the number of processors.

169

6.4. Libraries with Optimization Capabilities

speedup w.r.t. sequential program

Table 6.3. Running times and speedups of solveTS (naive skeleton program) andsolveTS dsf (optimized by DS fusion) against the number of processors. A speedup
is one with respect to a sequential program.
#processors
1
2
4
8
16
32
64
solveTStime (s) 117.03 69.92 40.54 20.21 10.14 4.97 2.79
speedup 0.53
0.89 1.54 3.08 6.13 12.52 22.28
solveTS dsf
time (s) 62.20 60.81 30.47 15.26 7.66 3.91 2.18
speedup1.00 1.02 2.04 4.08 8.12 15.90 28.49

30
25

solveTSsolveTS_dsf
1/2 linear

20
15
10
5
0
14 8

16

24 32
#processors

48

64

Figure 6.16. Speedups of solveTS (naive skeleton program) and solveTS dsf (optimized by DS fusion) with respect to the number of processors.
cessors, although the maximum speedup is limited to half of the number of processors. This is because the parallel implementation of scan essentially doubles its
computation with respect to that of sequential implementation.

6.4

Libraries with Optimization Capabilities

In this section, we will propose design and implementation of libraries with optimization capabilities, which can optimize naively written structured computations withknowledge of optimization theorems for skeleton programs. These libraries bringeasy use of the optimization theorems, to solve the problem of difficulty in applying
those theorems by hand, such as misjudgment of conditions, misimplementation ofcomplicated efficient implementation, and so on.
The library consists of two collections: one collection of the core objects toabstract various data structures, and the other collection of theorems to optimize
programs using the core objects. Each core object also carries out the whole com-170

6. Implementation of Skeletons and Optimizations

putation of the specific pattern on the structure, which is a quite natural design
to apply our theories in which the structures of the data and the computation areclosely related to each other. The core objects exploit the knowledge of theorems
in the other collection, to dispatch suitable efficient implementations to programs
naively written with the core objects.
We will show our idea with a development of a concrete library for nested reductions in new programming language Fortress [ACH+ 08]. The target computation of
the library is those discussed in Section 5.3.
In the example library, we will provide a collection of objects representing nested
data structures discussed in Section 5.3. For example, a code fragment to computethe maximum prefix sum is written with comprehensions as follows.
P
BIG MAX h h x | x ← px i | px ← prefixes xs i
Here, all prefixes are abstracted by the core object prefixes xs, in which the function
prefixes takes the input list xs and returns the core object. This core object canbe simply seen as aP
nested list of prefixes. The inner comprehension h x | x ← px iwith the reduction
computes a sum of each prefix px , and the outer h . . . | px ←prefixes xs i with the reduction BIG MAX (the maximum) takes the maximum of sums.
The example library is equipped with the theorems in Section 5.3, to optimize
those programs using the core objects. For example, the library applies Theorem 5.33 to the above program to execute it with the provided efficient implementation, since the program satisfies the application condition of the theorem.
Equipped with the collection of theorems, the library enables us to exploit knowledge of theorems easily, without applying theorems by hand. This clearly reduces
human errors such as misjudgment of conditions, misimplementation of complicatedefficient implementation, and so on.
In the rest of this section, we will first introduce the general design of libraries
with optimization capabilities. Then, to proceed to the example library for nested
reductions in Fortress, we will introduce the programming language Fortress. After
that, we will show design and implementation of the library for nested reductions.

6.4.1

General Design of Growable Optimizing Libraries

Figure 6.17 shows the general design of our growable optimizing libraries.
The point of the design of libraries is the two collections: One collection of thecore objects is provided to describe naive structured programs of specific patterns,
and the other collection of theorems is given to provide efficient implementations
under specific conditions on parameters of the computations.
The core objects represent the target data structures of computations, and perform the whole computations of specific patterns on the data structures. The core
object should handle the whole computation completely. This is a requirement
for exploiting knowledge of our theories because the computation structures in ourtheories are very closely related to the structures of target data.

171

6.4. Libraries with Optimization Capabilities

Collection of the core objects
for naive description of programs

Growable Optimizing Library

Collection of theorems
for optimization of programs

×

coreObj1 .compute(f1 , . . . , fk )Thm1 : (Condition1 , EfficientImpl1 )coreObj2 .compute(f1 , . . . , fk )Thm2 : (Condition2 , EfficientImpl2 )coreObj3 .compute(f1 , . . . , fk )interface(f1 , . . . , fk )coreObj4 .compute(f1 , . . . , fk )..
.

Growing◦

Thm3 : (Condition3 , EfficientImpl3 )Thm4 : (Condition4 , EfficientImpl4 )for i = 1, 2, . . .
if Conditioni (f1 , . . . , fk ) thenreturn EfficientImpli (f1 , . . . , fk )Thm5 : (Condition5 , EfficientImpl5 )..
.

Growing

A user program..
.
r = coreObj4 (⊕, ⊗, p, . . . , f )..
.

Figure 6.17. Two-collection design of growable optimizing libraries. Libraries optimize programs (specifications) naively written with the core objects. Each core
object carries out the optimization in the interface method, checking application conditions of theorems and executing the computation with efficient implementations
provided by applicable theorems. This structure enables easy use of knowledge oftheorems.

172

6. Implementation of Skeletons and Optimizations

For example, a list may be the core object, since it is a data structure, andit can handle the computation (homomorphism); the computation structure of thehomomorphism on the list is completely specified by the list. Actually, this idea isused in Fortress to provide simple computation pattern on data structures.
The computation on the core objects is carried out within an interface method
compute(f1 , . . . , fk ), which has some parameters of the computation such as functions, operators, predicates, and so on. For example, the homomorphism on a list
has two parameters.
The core objects optimize the computations on their representing data structures,
exploiting the collection of theorems. They dispatch suitable efficient implementations to the computations on the data structures themselves, when they find sometheorems are applicable to the computation with the given parameters.
The requirement that the core objects can handle the whole computation onthem is necessary to optimize the whole computation flexibly. The core objects
optimize the execution of the computation by replacing the naive algorithm to compute the result with efficient implementation. Thus, they have to handle the whole
computation.
The check on application conditions of theorems can be easily implemented by
the check on types. Since our theories formalize computation patterns as higher
order functions, the application conditions of theorems are usually described withwhether the parameters have specific mathematical properties or not. Also, thetest of possession of mathematical properties can be easily implemented by checking types, once we annotate the parameter objects with specific types to represent
mathematical properties. Therefore, the check on application conditions can be
implemented by the check on types.
Finally, it is worth noting that the library grows up by extending the collections.
The expressiveness is grown by enriching the collection of core objects. The powerof optimization is grown by enriching the collection of theorems.
This is our proposing design of growable optimizing libraries.

6.4.2

Parallel Programming Language Fortress

In this section we will review new parallel programming language Fortress [ACH+ 08].
The review starts at overviews, and proceeds to various expressions and the typesystem of Fortress.
Brief Overview of Fortress
Fortress is a new ambitious programming language being developed by Programming Language Research Group of Sun Microsystems Laboratories and its open
source community. Some interesting features are its default support for easy useof parallelism, its philosophy of a growing language, and its programming notation
closer to math.

6.4. Libraries with Optimization Capabilities

173

Fortress is a parallel language with high productivity for various parallel machines, supporting parallelism at several levels. The highest level support is expressions evaluated in parallel. The next level includes loops, reductions, and comprehensions. The lower level provides parallel code regions, and explicit multithreading.
In this thesis, we only use the high level support for parallelism.
Fortress is a growing language, i.e., grows over time, since we cannot build a
big language all at once. Observing this growing language philosophy, Fortress puts
as many language features as possible into libraries rather than the compiler, andsupports extensible syntax and many kinds of operators for DSLs. We also observe
the philosophy, and make a library with optimization capabilities.
Fortress supports notation closer to math. For example, it supports juxtaposition
notation so that we can write 2x to double x instead of 2 * x in the usual languages.
Also, we can write a function application without parenthesis, e.g., sin x instead ofsin(x). Fortress’ notation supports also many
operators, and various
P mathematical2comprehensions.
Therefore,
we
can
write
hx
|
x
←
xsi
to
take a square sum ofP
xs, in which
is an operator to take summation (a reduction with +) and h. . .i isa comprehension notation.
Expressions Evaluated in Parallel
Let’s consider the following tuple expression, in which f is a function and ei s arecertain expressions.
(f (e1 , e2 ), e3 + e4 )Fortress may evaluate elements of a tuple in parallel. Thus, sub expressions f (e1 , e2 )and e3 + e4 may be evaluated in parallel. In addition, arguments of a function (e1and e2 ), and operands of an operator (e3 and e4 ) may be evaluated in parallel,
respectively.
Expressions to Structure Programs
Let’s see some small programs to compute Fibonacci numbers.
The first program defines a recursive function fib.
fib(n : Z32) : Z32 = if n ≤ 1 then 1
else (f1 , f2 ) = (fib(n − 1), fib(n − 2))
f1 + f2
end
The function fib takes the input n of type Z32 (32-bit integer), and returns theresult (the nth Fibonacci number) of type Z32. The type of the input argument
follows the name n of the input after a colon, and the type of the result follows theparenthesis enclosing the input arguments. The function body is the ‘if’ expression,
which returns the last expression of the ‘then’ part when the condition holds, andotherwise returns the last expression of the ‘else’ part. Thus, the body return 1

174

6. Implementation of Skeletons and Optimizations

when n ≤ 1, and otherwise returns the sum f1 + f2 , i.e., the sum of the results of
the recursive calls fib(n − 1) and fib(n − 2) that are bound to variables f1 and f2
by the binding operator =. Types of f1 and f2 are not written in the code, and will
be inferred by Fortress. It is worth noting that an ‘if’ expression without the ‘else’part is allowed.
The following program defines function fibL to compute Fibonacci numbers withlinear cost. Its argument type and result type are omitted here, and will be inferred
automatically by Fortress.
fibL(n) = do (∗ defines a local function sub ∗)sub(m) = if m ≤ 1 then (1, 1)
else (f1 , f2 ) = sub(m − 1)
(f1 + f2 , f1 )end (∗ end of function sub ∗)(fst, snd ) = sub(n)fstend
The function body is the ‘do’ expression that executes its enclosing expressions
sequentially, and returns the result of the last expression. Therefore, the function
fibL returns the value of the variable fst, which is computed by the preceding expression that calls the local function sub computing the pair (fib(n), fib(n − 1)). Fortressallows declaration of local functions like sub. A comment in Fortress programs is
enclosed with (* and *).
We can also use a ‘while’ expression to compute Fibonacci numbers by a loop as
follows.
fibLW (n) = do
(f1 , f2 , m) : (Z32, Z32, Z32) := (1, 1, 1)
while m < n do (f1 , f2 , m) := (f1 + f2 , f1 , m + 1) end
f1
end
A ‘while’ expression evaluates its body (enclosed with do-end) repeatedly while thecondition holds. The pair (f1 , f2 ) holds (fib(m), fib(m − 1)) during the iterationin this program. The variable declaration uses operator := to assign their initialvalues, which indicates that the variables are assignable (updatable). Since we haveto use the operator := to update such assignable variables in Fortress, the body uses
the operator to update the variables with their new values, instead of the binding
operator =.
We can break an iteration of ‘while’ by an ‘exit-with’ expression. The following
program code replaces the ‘while’ loop followed by the expression f1 in the previous
program.

6.4. Libraries with Optimization Capabilities

175

label whileLoop
while true do
if m = n then exit whileLoop with f1 end(f1 , f2 , m) := (f1 + f2 , f1 , m + 1)
end
end whileLoop
The ‘exit-with’ expression exits from a block labeled by the ‘label’ expression, andreturns the expression following the ‘with’ as the result of the labeled block. Thus,the ‘exit-with’ expression of the above program exits the whileLoop block with thevalue of f1 .
Expressions to Branch on TypeThe next program prints the type of the given argument, using ‘typecase’ expressionthat branches according to types.
printType(x) = typecase x of
Z32 ⇒ println(“x is an integer.”)String ⇒ println(“x is a string.”)else⇒ println(“x is unknown type.”)end
The ‘typecase’ branches to the first case of which specified type matches withthe type of the given argument x . Therefore, the program prints “x is an integer.”when x has type Z32, and it prints “x is a string.” when x is a string. Otherwise“x is unknown type.” is printed. Of course, the dispatching based on types can also
be implemented with overloaded functions.
Fortress Type System
The types in Fortress consists of traits and objects. Traits are like interfaces inJava [GJSB05], but may contain code. Objects may have methods and fields, while
traits may not have fields. In that sense, objects are the actual data. The name ofa trait or an object may be used as a type. Traits and objects may extend multiple
traits, but may not extend any objects. In other words, objects are the leaves of
the hierarchy. Traits, objects, and methods may be parameterized. The parameters
may be types or compile-time constants.
Figure 6.18 shows some Fortress codes of traits and objects, of which functionalities will be explained in Section 6.4.3. Trait Generator is parameterized with
type E (enclosed with J and K), and has methods generate and reduce. The method
generate is also parameterized with another type R, and takes two arguments oftypes ReductionJRK (defined below) and E → R. Here, E → R is the type offunctions from E to R, and functions are first-class in Fortress. The other method
reduce has its body definition. Trait Reduction is parameterized with type L, and176

6. Implementation of Skeletons and Optimizations
trait GeneratorJEK
generateJRK(red : ReductionJRK, body: E → R): R
reduce(r: ReductionJEK) : E = generateJEK(r, fn (e : E) ⇒ e)end
trait ReductionJLK
empty(): Ljoin(a: L, b: L): L
end
trait SomeCommutativeReduction end
object SumReduction
extends { ReductionJNumberK, SomeCommutativeReduction }empty(): Number = 0
join(a: Number, b: Number): Number = a + b
end

Figure 6.18. Traits for generators and reductions.
Computing a reduction along with the tree

Generator h1, 2, 3, 4, 5, 6, 7, 8i⊕

Invocation of the interface method
⊕
4

8

3
1

generate(⊕, f )

⊕
⊕

7

2

5

f1

6

⊕
f4

⊕

f3

⊕

f2

f5

associativity

f8
f7

f6

associativity⊕⊕

generate(⊕, f )
1

2

3

4

5

6

7

8

⊕

⊕
f1

⊕
f2

f3

⊕
f4

f5

⊕
f6

f7

f8

Figure 6.19. Generators abstract parallel computation where a set of elements (data)is generated and then consumed by a reduction. A generator forms an abstract
computational tree. The associativity of reduction operators plays an importantrole in balancing the parallel computation.
has two (abstract) methods without their bodies. Object SumReduction extends
two traits Reduction with the type parameter L = Number (a trait for numbers)
and SomeCommutativeReduction. It also defines bodies of the abstract methods.

6.4.3

Generators for Reductions in FortressThis section reviews the concept of generators in Fortress, which is the most important idea of high-level parallelism in Fortress. We will also shows its implementation
briefly.
Basically, generators are objects that can generate a set of elements, and perform
parallel reduction on them. In other words, generators can be seen as the combination of list data structures and list homomorphism on them. Thus, our developed
theories can be applicable for them.

6.4. Libraries with Optimization Capabilities

177

Later in the next section, we will extend generators to handle nested computation
on nested structures.
Concept of Generators in Fortress
Generators in Fortress [ACH+ 08] are objects to abstract high-level parallel computation, in which a set of elements are generated and then consumed by a reduction
like a summation. A generator is a container holding a set of elements, and can beseen as a higher order function in the sense that it provides a pattern of computation with function parameters. Actually, the computation pattern is the same aslist homomorphism in Chapter 2.
Figure 6.19 shows how a generator abstracts such parallel computation. A
generator forms an abstract tree structure for parallel computation, in which elements are held in leaves connected by internal nodes. For example, generator
h1, 2, 3, 4, 5, 6, 7, 8i holds eight elements on the tree as shown in the top left of Figure 6.19, in which elements 1, 2, 3, and 4 are held in the left sub-tree and the others
are in the right sub-tree.
It is easily seen that the abstract structure is the same as the structure of joinlists. Also, the structure is the same as the computation structure of homomorphism.
A generator receives the pair of an associative binary operator and a function
through the interface method generate, to perform the computation along with theabstract tree structure. The abstract tree structure is filled with the operator andthe function, to become a concrete computational tree, in which a leaf becomes anapplication of the function to generate a new element, and an internal node is filledwith the binary operator. For example, the top right tree in Figure 6.19 shows one
filled with the pair (⊕, f ). The computation performed through generate(⊕, f ) ofgenerator h1, 2, 3, 4, 5, 6, 7, 8i results in f 1 ⊕ f 2 ⊕ f 3 ⊕ f 4 ⊕ f 5 ⊕ f 6 ⊕ f 7 ⊕ f 8,
which is the same as homomorphism.
Changing the operator and the function to be supplied, we can compute various
results from a generator. For example, the sum of squares of the elements are
obtained by the pair of addition operator + and function f (x) = x2 . Also, a productof all elements are computed by the product operator and the identity function.
A generator executes the computation in parallel. For example, two sub-trees in
the computational tree shown in the top right of Figure 6.19 are executed in parallel, because their computations are completely independent. This parallelization
corresponds to putting parenthesis as (((f 1 ⊕ f 2) ⊕ f 3) ⊕ f 4) ⊕ (((f 5 ⊕ f 6) ⊕f 7) ⊕ f 8).
The parallelism of the computation is completely controlled by a generator, inwhich the associativity of operators plays an important role in balancing the parallel
computation. For example, the abstract tree in the top left of Figure 6.19 can bechanged to the tree in bottom left of the figure, so that the grandchild sub-trees canbe evaluated in parallel evenly. The computation on the changed abstract tree shownin the bottom right results in the same value as the previous computation of the178

6. Implementation of Skeletons and Optimizations
generateJRK(red : ReductionJRK, body: E → R): R =if s ≤ 0 then red .empty()
else loop ′ (lo : E, hi : E): R =if lo = hi then body(lo)else (∗ Identify power-of-2 boundary in region, split there ∗)split = partitionL((lo BITXOR hi ) + 1)
mid = hi BITAND (BITNOT(split − 1))
red .join(loop ′ (lo, mid − 1), loop ′ (mid , hi ))endloop ′ (b, b + s − 1)end

Figure 6.20. Simple parallel implementation of the method generate.
top right tree, since the operator has associativity, i.e., (((f 1 ⊕ f 2) ⊕ f 3) ⊕ f 4) =((f 1 ⊕ f 2) ⊕ (f 3 ⊕ f 4) and so on.
Comprehension notation is provided in Fortress for concise use of generators.
Roughly speaking, the following comprehension with a reduction is equivalent to
the invocation of generate(⊕, f ) on a generator g.
L

hf a | a ← gi

Here, the comprehension hf a | a ← gi specifies generation of elements by
Lapplying
the function f to each element a held in the generator g, and big operator
specifies
a reduction with ⊕ on the generated elements.
For example, to take the summation of squares of list h1, 2, 3, 4i, we can use thefollowing comprehension.
P

ha2 | a ← h1, 2, 3, 4iiThis comprehension is equivalent to the following skeleton program.
reduce (+) (map (λa.a2 ) [1, 2, 3, 4])Comprehensions
in Fortress can handle predicates to filter generated elements.
P
For example, ha | a ← x, a > 0i results in a sum of positive elements by using predicate a > 0. This is equivalent to skeleton program reduce (+) (filter (λa.a > 0) x).
However, we omit handling of predicates for simplicity, until Section 6.4.5.
Implementation of Generators in Fortress
A generator in Fortress is an object extending trait Generator with the interface
method generate. Its core definition is shown in Figure 6.18. The method generatetakes an associative binary operator red and a function body, to perform the generation and the reduction.

179

6.4. Libraries with Optimization Capabilities⊕cprefixes
ha, b, ci

a

a

b

b

⊕

generate(⊕, generate(⊗, f ))

⊗
⊗
fa

fa

b

Dependency among them is ignored.
c

a

a

fc
fb

⊗
fa

fb

Their independent computation causes inefficiecy.

Nested reduction on prefixes of ha, b, ci by the current generators.
Nested reduction on prefixes of ha, b, ci by generators-of-generators.

prefixescagenerate2 (⊕, ⊗, f )Efficient implementation exploiting the dependency andproperties of ⊕ and ⊗.

b

Figure 6.21. Computation of a nested reduction by the current generators and by
the GoGs.
An associative binary operator is enclosed in an object extending trait Reduction.
Trait Reduction has method join to provide an associative binary operator, andmethod empty to provide the identity of the operator.
For example, the usual addition operator + is enclosed with object SumReduction
shown in Figure 6.18, in which the method join adds two operands by the operator+, and the method empty returns 0, i.e., the identity of +.
Figure 6.20 shows an example parallel implementation of generate method ofgenerator (b # s) that produces a sequence of s numbers from b, i.e., b, b + 1, . . . , b +s − 1. When s ≤ 0, i.e., the generator produces no elements, the method returns
the identity red .empty(). Otherwise, it forms an abstract binary tree by recursive
calls of the local function loop ′ that performs partial computation on lo, lo + 1, . . .,hi. In the case of lo = hi , i.e., a leaf of lo, it generates a new element by body(lo).
In the other case, i.e., an internal node, it combines the results loop ′ (lo, mid − 1)and loop ′ (mid , hi ) of sub-trees by the given associative binary operator red .join.
The results of sub-trees are computed in parallel, since operands are computed inparallel by default in Fortress. The structure of the abstract computational tree
varies according to the number of elements to be generated. The associativity of thegiven operator is necessary to obtain a well-defined result of the computation.
Finally, we briefly mention desugaring of comprehensions in Fortress. Each useof comprehension is desugared into invocation of the method generate of a generator
by the following desugaring algorithm DS.
L
DS( hbody | x ← g, gsi)
L=g.generate(⊕,
fn
x
⇒
DS(
hbody | gsi))
L
DS( hbody |i)=bodyP
For example, the code hx2 | x ← xsi to compute a sum of squares is desugared by
the compiler into the following invocation of generate. Here, the reduction operator

180

6. Implementation of Skeletons and Optimizations

object SumReduction is given by the operatorP.

xs.generate(SumReduction, fn x ⇒ x2 )6.4.4

Generators-of-generators: the Core of Nested Reductions

We will introduce the concept of generators-of-generators (GoGs for short) to represent nested data structures of specific patterns, which is the core objects for nested
reductions. Before introducing GoGs, we will first point out the problem of using
the current simple generators.
Problem of Using the Simple Generators
The requirement for the core objects is that they can handle the whole computationon them, i.e., the whole computation of nested reductions, so that they can optimize
the whole computation flexibly. They optimize the execution of the computation byreplacing the naive algorithm to compute the result with efficient implementation.
Thus, they have to handle the whole computation.
However, the current generators cannot satisfy the requirement. This is mainly
because they are designed to handle only flat reductions on them. Even though theirgenerating elements have dependencies on each other, and thus the reductions onthem have the induced dependencies, these dependencies are completely ignored andthe whole computation of nested reductions cannot be handled with one generator.
Consider computing nested reductions on dependent data structures such asprefixes of a sequence. With the existing generators, nested reductions on prefixesof generator xs = ha, b, ci with two operators ⊕ and ⊗ are computed by the following
code.
L N
h h f xs i | i ← (0 # s) i | s ← (1 # (|xs|)) i

Here, |xs| is the size of xs. The top half of Figure 6.21 illustrates the computation,in which leaves of the big abstract computational tree are (virtually) abstract treesof prefixes hai, ha, bi, and ha, b, ci.
The problem is that the outer tree generates the inner abstract trees independently in its leaves, and thus inner computations are independently executed in theleaves, even though the inner trees have the dependency that they are prefixes of
one generator xs.
To solve the problem, we will introduce generators-of-generators, which abstracts
nested data structures.
Generators-of-Generators to Capture the Whole Computation of Nested
Reductions
To solve the problem of the current generators ignoring inner generators, we will
introduce generators-of-generators, which abstracts nested data structures. They

6.4. Libraries with Optimization Capabilities

181

capture dependencies among generated generators such as prefixes, and abstract
the whole structure of a nested reduction on them. Therefore, we can have a big
chance to perform efficient computation exploiting the dependency and propertiesof operators.
GoGs are more-informed generators for efficient computation of nested reductionsof the following form.
L N
h h f y | y ← ysi | ys ← gg iHere, gg is a GoG to produce a set of dependent generators (yss). In the rest of
this chapter, we will especially focus on a GoG such that a generated generator is asubset of a given generator.
The computation of a nested reduction with a GoG is illustrated in the bottomof Figure 6.21. A GoG is also a generator, and forms a similar nested abstract
computational tree. However, a GoG abstracts the whole structure of a nested
reduction by one object, while the existing generators do by multiple objects (i.e.,the outer generator and the inner generators).
GoGs have the new interface method generate2 to execute nested reductions withthe given operators, so that they can exploit mathematical properties between thegiven operators and the dependencies among inner generators for optimization basedon theorems. The new interface receives two reduction operators at the same time,
while the computation by the existing generators supply these operators separately
to the outer generator and to the inner generators. This point is important, because
GoGs are often required to handle these two operators simultaneously to exploit
knowledge of theorems.

6.4.5

Design and Implementation of the Optimizing GoG
Library

We will implement the growable optimizing library for nested reductions, based on
the general design in Section 6.4.1.
The main features of the library are two folds.
• It provides a set of GoGs so that users can easily write programs for nested
reductions with various patterns of dependencies. As demonstrated later, various interesting applications can be clearly specified with the GoGs the libraryprovides.
• It, like other optimization techniques based on simple calculation rules [GLJ93,
OHIT97], implements the optimization lightweightly deep analysis of programcodes and can be implemented as a Fortress library. The simplicity and thenontrivial improvement are owing to the use of calculation theorems and thepowerful dispatch mechanism of Fortress.
Our optimizing library for nested reductions has been integrated into the current
Fortress interpreter [For08].

182

6. Implementation of Skeletons and Optimizations
Collection of generators-of-generatorsfor describing specifications

A user programLN
h hf y | y ← ysi | ys ← prefixes xsi
LN
h hf y | y ← ysi | ys ← suffixes xsi
LN
h hf y | y ← ysi | ys ← segments xsi

Growable Optimizing Library×

Thm1 : (Condition1 , EfficientImpl1 )generate2 (⊕, ⊗, f )◦Thm2 : (Condition2 , EfficientImpl2 )Thm3 : (Condition3 , EfficientImpl3 )LN
h hf y | y ← ysi | ys ← subseqs xsi..
.

Collection of theorems
for optimizing reductions

Thm4 : (Condition4 , EfficientImpl4 )for i = 1, 2, . . .
if Conditioni (⊕, ⊗, f ) thenreturn EfficientImpli (⊕, ⊗, f )Growing

Thm5 : (Condition5 , EfficientImpl5 )..
.

Growing

Figure 6.22. Two collections form a growable optimizing library. It optimizes specifications using knowledge of theorems on them.
Figure 6.22 shows the structure of the library, which is an instance of the designin Section 6.4.1. The core objects in the library are GoGs.
The library supports nested reductions of the following form with various GoG
gg.
L N
h h f y | y ← ysi | ys ← gg i

The behavior of programs written with our library is as follows. A nested reduction written a GoG is desugared into an invocation of the interface method generate2
with the given parameters, i.e, two reduction operators and a function applied toeach element. Then, the method generate2 checks whether the given parameterssatisfy the application conditions of theorems stored in the library. Once the condition of a theorem is found to be satisfied, the computation of the nested reduction
is carried out with the efficient implementation the theorem provides. If no theorem
is available, the nested reduction is carried out with the naive implementation, in
which all dependent generators are actually generated independently and inner reductions on them are performed also independently. This mechanism of dispatching
implementation enables users to exploit knowledge of theorems implicitly.
The optimization mechanism by dispatching needs to check mathematical properties of operators and functions to utilize calculation theorems. Generally, it is very
difficult to prove a mathematical property at compile-time or, of course, run-time.
Thus, we assume that operators and functions are beforehand annotated about their
specific properties with specific types. The dispatching and checking mechanisms
are explained in Section 6.4.5.
Mechanism of Growable Optimizing Library
We show the details of the implementation of our library, with examples of a GoG
for prefixes.
We will first show the mechanism with the implementation for one-dimensional
data structures, i.e., for lists. After the explanation of the mechanism, we willshow the implementation for two-dimensional arrays, i.e., the implementation forthe bide-tree homomorphism.

6.4. Libraries with Optimization Capabilities

183

trait Generator2J E K extends {GeneratorJ GeneratorJ E K K }getter seed () : GeneratorJ E K
generate2 J R K(q : ReductionJ R K, r : ReductionJ R K, f : E → R) : R
theoremsJRK() : ListJ((ReductionJ R K, ReductionJ R K, E → R) → Boolean,(ReductionJ R K, ReductionJ R K, E → R) → R)K = h inaiveImpl J R K(q : ReductionJ R K, r : ReductionJ R K, f : E → R) : R =generateJRK(q, (fn (x) ⇒ x.generateJRK(r, f )))end

Figure 6.23. Base trait of GoGs
Base Trait of GoGs: Generator2
Figure 6.23 shows the base trait Generator2 of all GoGs. Since a GoG is a generator
producing generators, it extends GeneratorJGeneratorJEKK, although it does notnecessarily actually do the generation.
Since a GoG produces a set of generators that are subsets of a given generator,
trait Generator2 has a getter seed () to hold the given generator. The given generator
is used by efficient implementations to exploit the parallelism in the generator.
Also, it is used by naive implementations to produce the naive nested computation
structure.
The interface method generate2 receives two objects of reduction operators anda function to be applied to every element, to perform the nested reduction withthe given parameters. It implements the dispatching mechanism to optimize nested
reductions using calculation theorems, which is explained in Section 6.4.5.
Knowledge of theorems on a GoG is stored as the list theorems of pairs of functions. Each pair of the list consists of (1) a function to check the application condition of a theorem, and (1) a function to execute efficient implementation provided by
the theorem. These functions take the same arguments of the method generate2 , andreturn a Boolean value of satisfiability of the condition and the result of the nested
reduction with the given parameters, respectively. The list is empty by default, andshould be enriched by library implementers to utilize knowledge of theorems (seeSection 6.4.5).
Method naiveImpl provides the naive implementation to perform the nested
reduction when no theorems is available. It invokes the method generate with theouter reduction operator and a function that invokes the method generate of an
inner generated generator with another operator and the function, which is thenaive execution shown in the top half of Figure 6.21.
For example, Figure 6.24 shows the least implementation of a GoG for prefixes,
which corresponds to inits in Section 5.3. The least requirement for defining a GoG
is to implement the getter seed () and the method generate, which is an abstract
method of trait Generator. The object Prefixes receives a generator of which prefixes
are to be generated, and stores it in the getter seed (). The implementation of themethod generate actually produces all prefixes by function prefixesImpl , and then184

6. Implementation of Skeletons and Optimizations
object PrefixesJ E K(g : GeneratorJ E K) extends Generator2J E K
getter seed () : GeneratorJ E K = g
generateJ R K(r : ReductionJ R K, body : GeneratorJ E K → R) : R= prefixesImpl J E K(g).generateJ R K(r, body)end
prefixesImpl J E K(x : GeneratorJ E K) : GeneratorJ GeneratorJ E K K= h h xi | i ← (0 # s) i | s ← (1 # (|x|)) i

prefixesJEK(g : GeneratorJEK) : PrefixesJEK = PrefixesJEK(g)Figure 6.24. The least implementation of a GoG for prefixes.
generate2 JRK(q : ReductionJRK, r : ReductionJRK, f : E → R) : R= do ths = theoremsJRK()i : Z32 := 0label dispatchingLoop
while i < |ths| do(condition, efficientImpl ) = ths iif (condition(q, r, f )) then(∗ use efficient implementaion ∗)exit dispatchingLoop with efficientImpl (q, r, f )end
i += 1end
naiveImpl J R K(q, r, f )(∗ use naive implementaion ∗)end dispatchingLoop
end

Figure 6.25. The implementation of the dispatching mechanism.
invokes the method generate of the generated prefixes. This is the required behaviorof the GoG as a Fortress’ generator to produce prefixes.
Function prefixesImpl implements the actual generation of prefixes, in which alist of prefixes is generated by the comprehension, where hxi | i ← (0 # s)i is the sth
prefix of x . The last function prefixes is a short cut to make a GoG for prefixes of
the given generator.
Dispatching Mechanism in Method generate2
Method generate2 implements the dispatching mechanism to optimize nested reductions with calculation theorems, which is the most important function of the library.
Figure 6.25 shows implementation of the method, which takes the parameters of thenested reductions, i.e., two objects q and r of reduction operators and a function fapplied to every element.
The main loop in method generate2 checks each application condition conditionof theorems stored in the list theorems. When the condition of a theorem is satisfied

6.4. Libraries with Optimization Capabilities

185

trait DistributesOverJEK end
distributesJQ, RK(q : Q, r : R) : Boolean =typecase (q, r) of(Q, DistributesOverJQK) ⇒ trueelse⇒ false
end

Figure 6.26. A trait for annotation about distributivity and a function to check the
distributivity.
object SumReduction
extends { ReductionJNumberK, SomeCommutativeReduction,
DistributesOverJMaxReductionK, DistributesOverJMinReductionK }empty(): Number = 0
join(a: Number, b: Number): Number = a + b
end

Figure 6.27. An annotated reduction object for summation.
by the given parameters q, r, and f , the dispatching loop carries out the nested
reduction with the efficient implementation efficientImpl of the theorem, and exits
the dispatching loop with the computed result. If there is no theorem available, thenaive implementation naiveImpl is used to get the result. The current dispatching
loop assumes that a theorem appearing earlier provides more efficient implementation.
The check on application conditions is performed by checking annotations ontypes of the objects. For example, Figure 6.26 shows a trait DistributesOverJEK toannotate a reduction object about its distributivity. When an operator represented
by an object of type R has distributivity over another operator represented by anobject of type Q, R extends the trait DistributesOverJQK to inform the library of
its possessing the property. Then, a function distributes checks possession of the
distributivity by using typecase expression. Other properties can be annotatedsimilarly using traits.
Figure 6.27 shows reduction object SumReduction annotated with the trait forthe distributivity. Since the addition operator distributes over the maximum operator and the minimum operator, the object extends DistributesOverJMaxReductionKfor distributivity over the maximum, and DistributesOverJMinReductionK for distributivity over the minimum.
Using such annotations on objects, we can successfully implement knowledge oftheorems in the library. Figure 6.28 shows example implementation of the combination of Theorem 5.33 and Lemma 5.31 on prefixes. Its translation for comprehension
notation is as follows.

186

6. Implementation of Skeletons and Optimizations
conditionDistributiveJ R K(q : ReductionJ R K, r : ReductionJ R K, f : E → R) : Boolean= distributes(q, r)
efficientImplDistributiveJ R K(q : ReductionJ R K, r : ReductionJ R K, f : E → R) = dojoin(x : (R, R), y : (R, R)) : (R, R) = do (i1 , s1 ) = x(i2 , s2 ) = y(q.join(i1 , r.join(s1 , i2 )), r.join(s1 , s2 ))end
zero1 = (q.empty(), r.empty())
wrap(a : R) : (R, R) = (a, a)
unwrap(a : (R, R)) = do (r1 , r2 ) = a; r1 end
unwrap(seed ().generateJ(R, R)K(makeReductionJ(R, R)K(join, zero1 ), wrap ◦ f ))endtheoremsJRK() = h (fn (r, q, f ) ⇒ conditionDistributiveJ R K(r, q, f ),fn (r, q, f ) ⇒ efficientImplDistributiveJ R K(r, q, f )) i

Figure 6.28. A method to check the condition of Lemma 6.4 and a method implementing the provided efficient implementation. A pair of the two methods is added
to the list theorems in the GoG object of prefixes, so that the library can exploit its
knowledge.
Theorem 6.4. Provided that ⊗ distributes over ⊕, the following equation holds.
LN
h hf yJ| y ← ysi | ys ← prefixes xsi= fst2 ( p h (f x, f x) | x ← xsi)
where (i1 , s1 ) ⊙p (i2 , s2 ) = (i1 ⊕ (s1 ⊗ i2 ), s1 ⊗ s2 )fst2 (a, b) = a
The library implements the theorem as the pair of methods: conditionDistributive
to check its application condition, and efficientImpDistributive to perform its providing efficient implementation. Since the condition is that the operator ⊗ is distributive over ⊕, the method conditionDistributive uses the function distributes to knowwhether the operator has the distributivity. The method efficientImpDistributive
implements the efficient implementation straightforwardly, in which it uses direct
invocation of method generate instead of the comprehension, and the function
makeReduction encloses the new operator (join) and its identity (zero1 ) in an object. Finally, to use the knowledge of the theorem in the library, the pair is added
to the list theorems.
It is easily seen that the same functionality to dispatch efficient implementation
based on theorems is implementable for other domains of computations.
Extension of Desugaring Process
We extend the desugaring process of comprehensions in Fortress to handle GoGs innested reductions. The following rules are added to the existing desugaring algo-

6.4. Libraries with Optimization Capabilities
Target specification of nested reductions
L N
h hf y | y ← ysi | ys ← prefixes xsi
L N
LhNhf y | y ← ysi | ys ← suffixes xsi
h hf y | y ← ysi | ys ← segments xsi
L N
h hf y | y ← ysi | ys ← prefixes xs, p ysi
L N
h hf y | y ← ysi | ys ← suffixes xs, p ysi
L N
h hf y | y ← ysi | ys ← segments xs, p ysi

187

Application condi- Efficient implementation.
tionJ⊗ distributes over ⊕ fst 2 ( p h(f x, f x) | x ← xsi)
J⊗ distributes over ⊕ fst 2 (Js h(f x, f x) | x ← xsi)
⊗ distributes over ⊕, fst 4 ( t h(f x,f x,f x,f x) | x ← xsi)
⊕ is commutativeJ⊗ distributes over ⊕, fst 4 ( p′ h(f x, f x, x, x) | x ← xsi)
p is relationalJ⊗ distributes over ⊕, fst 4 ( s′ h(f x, f x, x, x) | x ← xsi)
p is relationalJ⊗ distributes over ⊕, fst 6 ( t′ h(f x,f x,f x,f x,x,x) | x ← xsi)
⊕ is commutative, p
is relational

Table 6.4. A part of the collection of the composed theorems written in comprehension notation.
rithm.

LN
h hf x | x ← xsi | xs ← ggi) = gg.generate2 (⊕, ⊗, f )LDS( hbody | x ← gg, p x, gsi)
L
= DS( hbody | x ← gg.filter (p), gsi)
DS(

The first rule desugars a nested reduction with a GoG into an invocation of themethod generate2 of the GoG. This rule connects the comprehension notation tothe optimization mechanism of GoGs.
The second rule squeezes a filter with a predicate p into the preceding GoG gg,
to result in another GoG gg.filter (p) that generates only filtered generators of thosethe original produces. Each GoG has the method filter to take a predicate andreturns such another GoG, although it is not discussed in the implementation of
the GoG in the previous section for simplicity. The extension of the optimization
mechanism to involve predicates is straightforward.
It is worth noting that the beta specification of Fortress supports library-level
extensions of syntax for domain-specific languages. We would be able to use thefeature to implement the desugaring process completely in our library, although
the current extension is implemented in the interpreter. It would be noted thatthe desugaring process is independent from the optimization mechanism; it simply
supports easy description of nested reductions.
Growing the Library
The library can improve its expressiveness and optimization power by enriching thecollections of GoGs and theorems.
We have added GoGs corresponding to the generation functions shown in Section 5.3. Some functions have alternative familiar names: prefixes for inits, suffixes
for tails, segments for segs, subseqs for subs, and subrecs for rects ′ .
Also, we have added theorems (lemmas) shown in Section 5.3, in which sometheorems are implemented as combinations of two or more theorems. Some of implemented theorems are shown in Table 6.4.

188

6. Implementation of Skeletons and Optimizations
trait GeneratorATJEK
generateAT JRK(op : ReductionJRK, ot : ReductionJRK, body: E → R) : R
end

Figure 6.29. The base trait of generators for two-dimensional data structures. The
trait defines the interface method for the computation of the abide-tree homomorphism.
Implementation of GoGs for Two-dimensional Arrays
Now, we will briefly show the implementation of the abide-tree homomorphism in
GoGs for two-dimensional arrays.
Figure 6.29 shows the base trait of generators for two-dimensional data structures. The trait defines the interface method for the computation of the abide-treehomomorphism. The interface method takes two reduction operators: one is for thevertical direction and the other is for the horizontal direction.
Figure 6.30 shows concrete implementation of the abide-tree homomorphism, inwhich the index spaces are divided into smaller ones in both horizontal and verticaldirections to get the sub-results, and the sub-results are combined by the given
operators. The wrapper object is prepared to perform the abide-tree homomorphism
on usual arrays not extending the trait.
Figure 6.31 shows the base trait of GoGs for two-dimensional data structures.
The new interface method receives four reduction operators: the first two for theouter reduction, and the last two for the inner reductions. The trait has the list oftheorems, which is similar to the list in the trait Generator2, except for the numberof reduction operators.
The dispatching process is implemented similar to that of the method generate 2 .
Since comprehension notation for the abide-tree homomorphism is not preparedin Fortress, users are required to invoke the interface method directly to performthe abide-tree homomorphism. Fortunately, we can use the current comprehension
notation if we use the same operators in both directions. For example, the maximum rectangle sum problem is one example of such computation. In this case, theinterface method generate 2 is invoked, and we can dispatch efficient implementationin this method as shown in the previous section. It is an interesting task to design
comprehension notation for the abide-tree homomorphism.

6.4.6

Programming with the Library

We will show example programs written with the library.
Figure 6.32 shows an example Fortress program to compute the maximum psegment sum with the GoG library. The program imports the component Generator2
to use GoGs, and also imports List to use comprehensions. The main function run
first makes a one-dimensional array xs of 400 elements, and two relational predicates6.4. Libraries with Optimization Capabilities

189

object GeneratorATWrapperJE, nat b1 , nat s1 , nat b2 , nat s2 K(x : Array2JE, b1 , s1 , b2 , s2 K)
extends GeneratorATJEK
generateAT JRK(op : ReductionJRK, ot : ReductionJRK, body: E → R) : R = doloop ′ (lo1 : E, hi1 : E, lo2 : E, hi2 : E) =if lo1 = hi1 ∧ lo2 = hi2 then body(x[(lo1 , lo2 )])elif lo1 = hi1 thensplit2 = partitionL((lo2 BITXOR hi2 ) + 1)
mid2 = hi2 BITAND (BITNOT(split2 − 1))
ot.join(loop ′ (lo1 , hi1 , lo2 , mid2 − 1), loop ′ (lo1 , hi1 , mid2 , hi2 ))elif lo2 = hi2 thensplit1 = partitionL((lo1 BITXOR hi1 ) + 1)
mid1 = hi1 BITAND (BITNOT(split1 − 1))
op.join(loop ′ (lo1 , mid1 − 1, lo2 , hi2 ), loop ′ (mid1 , hi1 , lo2 , hi2 ))
else
split2 = partitionL((lo2 BITXOR hi2 ) + 1)
mid2 = hi2 BITAND (BITNOT(split2 − 1))
split1 = partitionL((lo1 BITXOR hi1 ) + 1)
mid1 = hi1 BITAND (BITNOT(split1 − 1))
op.join(ot.join(loop ′ (lo1 , mid1 − 1, lo2 , mid2 − 1), loop ′ (lo1 , mid1 − 1, mid2 , hi2 )),ot.join(loop ′ (mid1 , hi1 , lo2 , mid2 − 1), loop ′ (mid1 , hi1 , mid2 , hi2 )))endloop ′ (b1 , b1 + s1 − 1, b2 , b2 + s2 − 1)
end
end

Figure 6.30. Implementation of the abide-tree homomorphism with two operators.
The wrapper object is prepared to perform the computation on the usual arrays.
flat 4 and ascending (Section 5.3.3) by specifying their relations to the predicategeneration function relationalPredicate. Then, it computes the maximum flat 4 ascending-segment sum of xs, generaing all segments by the provided function segs,P
filtering the segments by the predicates, taking sums of segments by the reduction ,and taking the maximum of sums by the reduction with BIG MAX. Here, the program
explicitly specifies theP static type parameters JNumberK of
P the operators and thecomprehension, and JNumberKys is the abbreviation of JNumberKhy | y ← ysi.
It is worth noting that generators in Fortress is equipped with some fusion optimizations such that two consecutive maps are fused into one map, and two filter s
are also fused into one filter . Delaying their computations by making computation
objects instead of immediate execution of the computations, Fortress implements
those fusions at the library level.
Similarly, in arbitrary points of Fortress programs, we can write various comprehensions with GoGs to perform nested reductions. Table 6.5 lists comprehensions
written with
where ↓ is theQ
P GoGs for various applications,Q
P minimum operator, andys andys are abbreviations of hy | y ← ysi and hy | y ← ysi, respectively.
Nested reductions with GoGs can describe various applications including typicalproblems in the functional community [Bir87, Gor97, SHTO00, Bir01], such as the

190

6. Implementation of Skeletons and Optimizations

trait Generator2ATJEK extends { Generator2J E K, GeneratorATJGeneratorATJEKK }generate2AT JRK(op : ReductionJRK, ot : ReductionJRK, om : ReductionJRK, od : ReductionJRK, body: E → R) : R
theoremsAT JRK() : ListJ((ReductionJ R K, ReductionJ R K, ReductionJ R K, ReductionJ R K, E → R) → Boolean,(ReductionJ R K, ReductionJ R K, ReductionJ R K, ReductionJ R K, E → R) → R)K = h inaiveImplAT J R K(op : ReductionJRK, ot : ReductionJRK, om : ReductionJRK, od : ReductionJRK, f : E → R) : R =generateAT JRK(op, ot, (fn (x) ⇒ x.generateAT JRK(om, od , f )))end

Figure 6.31. The base trait of GoGs for two-dimensional data structures. The interface method receives four reduction operators: the first two for the outer reduction,and the last two for the inner reductions.
component ExampleProgram
import List.{. . .}import Generator2.{. . .}export Executablerun() : () = do
xs = arrayJNumberK(400).fill (fn (x : Z32) : Number ⇒ ⌊random(10) − 5⌋)f lat4 = relationalPredicateJNumberK(fn (a, b) ⇒ |a − b| < 4)ascending = relationalPredicateJNumberK(fn (a, b) ⇒ a < b)Pmpss = BIG MAX JNumberKhJNumberK JNumberKys | ys ← segs xs, f lat4 ys, ascending ys i
println(“the maximum flat4-ascending-segment sum of xs is ” mpss)

end
end

Figure 6.32. Ax example Fortress program to compute the maximum p-segment
sum with the GoG library.
maximum prefix sum problem in the introduction, and the maximum segment sum
problem that is a simplified problem to find a region of interest in a given sequence.
Also, use of predicates can control targets of the inner reductions, which further
broadens the application area of GoGs. The predicates descending and equiv usedin examples are relational predicates of relations xi−1 > xi and xi−1 = xi , respectively.
Most of the example comprehension shown above will be optimized at the runtime by executing the computation with efficient algorithms, although their naive
computations are inefficient.
One feature of writing programs with GoGs is its easiness of writing nested reduction with various dependencies in clear and uniform expression. Writing nested
reductions without GoGs usually requires explicit use of several or many indices generated by generators, which decreases productivity and maintainability of programs.
For example, a naive reduction on prefixes or suffixes uses two indices, and that ofsegments needs three. Moreover, writing nested reductions on all subsequences isvery difficult, because it requires infinite indices to write it in the same way.

6.4. Libraries with Optimization Capabilities
Label
MPS
MSP
MSS
PSS
pMPS

Fortress P
code with GoGs
BIG MAX hQ ys | ys ← prefixes xsi
BIG MIN hP ys | ys ← suffixes xsi
BIG
MAX h ys | ys ← segments xsi
P Q
h ys |Pys ← subseqs xsi
BIG MAX h ys | ys ← prefixes xs, flat4 ysi

pMSP

BIG MIN hQ

191

Description
The maximum of sums of prefixes
The minimum of products of suffixes
The maximum of sums of all continuous subsequences
The sum of products of all subsequences
The maximum sum of 4-flat prefixes in which every differenceof neighboring elements is less than 4.
The minimum product of descendingly ordered suffixes.

ys | ys ← suffixes xs,
descending ysi
The maximum sum of ascendingly ordered, 4-flat segments.
pMSS BIG MAX h ys | ys ← segments xs,
flat4 ys, ascending ysi
P Q
pPSS
The sum of products of subsequences made of equivalents.
h ys |Pys ← subseqs xs, equiv ysi
MRS BIG MAX h ys | ys ← subrecs xsi
The maximum rectangle sum.
P

Table 6.5. Example comprehensions with GoGs.
Another advantage of writing programs with GoGs is that it reduces the risk oflosing maintainability caused by hand-optimization. Performing optimization withhigh-level mathematical knowledge (i.e., calculation theorems) is not so hard forhumans, but doing it by hand poses the risk of losing maintainability. A programmer
cannot return an optimized program (i.e., an efficient implementation provided by atheorem) to the non-optimized program, to change the reduction operators to those
not satisfying the application condition, unless he knows the original program.
Therefore, GoGs are very useful for easy and clear development of programs fornested reductions.

6.4.7

Experiment Results

We measured the execution time of micro-benchmarks listed in Table 6.5 with andwithout the optimization. We used the current Fortress interpreter (release 3294
from the subversion repository) [For08] run on a PC with two quadcore CPUs
(Intel R Xeon R E5430, total 8 cores), 8GB memory, and Linux 2.6.24.
Table 6.6 lists measured execution time for several data sets. The sizes of input
sequence (array) xs of the data sets (S, M, L, X) are (600, 1200, 12000, 120000) for
MPS, MSP, pMPS and pMSP, (60, 120, 1200, 12000) for MSS and pMSS, (8, 16,—, —) for PSS and pPSS, (8 × 8, 16 × 16, 64 × 64, 128 × 128) for MRS. The tablealso lists asymptotic cost of the computation for the input of size n (the length of
the input lists, and the width and the height of input arrays).
The figures of S and M without optimization show that the naive implementations are actually inefficient since their asymptotic cost is more than linear. Also,
comparison of the results of M with and without optimization shows that the absolute execution time without optimization is much greater than that of efficient
implementations dispatched by the library. The figures for L and X with optimization show that asymptotic cost of dispatched implementation is actually linear for
MPS, MSP, MSS, pMPS, pMSP, and pMSS, and cubic for MRS. It is worth noting that codes dispatched by the library can achieve the same execution times as
hand-coded solutions to the problems, since the library dispatches the hand-coded
implementations given by calculation theorems, as long as they use generators for192

6. Implementation of Skeletons and Optimizations

Label
MPS
MSP
MSS
PSS
pMPS
pMSP
pMSS
pPSS
MRS

Without optimization
Asym.
S
M
cost
O(n2 )
6.55
16.4
O(n2 )
2.41
7.66
O(n3 )
2.50
13.7
O(2n )
0.829
18.8
O(n2 )
7.17
26.2
O(n2 )
4.88
18.0
O(n3 )
3.04
18.3
O(2n )
0.192
21.5
O(n6 )
5.52
615.12

With optimization
Asym.
M

L

X

O(n)
O(n)
O(n)
—
O(n)
O(n)
O(n)
—
O(n3 )

0.576
0.772
0.110
—
0.666
0.627
0.132
—
65.94

2.02
1.85
0.34
—
4.29
5.26
0.903
—
370.2

0.253
0.21
0.053
—
0.257
0.152
0.0389—3.00

Table 6.6. Measured execution time (in seconds) of nested reductions in Table 6.5
with/without optimization. The sizes of input sequence (array) xs of the data sets
(S, M, L, X) are (600, 1200, 12000, 120000) for MPS, MSP, pMPS and pMSP, (60,120, 1200, 12000) for MSS and pMSS, (8, 16, —, —) for PSS and pPSS, (8 × 8,
16 × 16, 64 × 64, 128 × 128) for MRS. The table also lists asymptotic cost of thecomputation for the input of size n (the length of the input lists, and the width andthe height of input arrays).
# of thm.
Time

1
3.55

4
3.18

16
4.3

64
9.6

256
28.7

1024
103

4096
395

Table 6.7. Overheads of the dispatching process for various numbers of theorems(in milli-seconds).
flat reductions. The results of PSS and pPSS with optimization are omitted since
the library currently has no theorem for them. These results shows the library’s
optimization is dramatically effective for efficient computation of various nested reductions.
We also measured overhead of the dispatching process in the method generate2 .
We used dummy GoGs such that each of them has the list theorems of l dummy
theorems. The condition of the dummy theorem checks distributivity and commutativity of operators (with functions in Figure 6.26) but returns always false. Thus,
the dispatching process checks the dummy condition l times, and fails in finding anapplicable theorem. It finally executes its dummy naive implementation that performs nothing. Therefore, we measured the execution time of the method generate2
of dummy GoGs as the time of dispatching process.
Table 6.7 lists measured time of dispatching process for various sizes of theorem
lists. The time of dispatching process is very small and ignorable against thatof nested reductions, unless too many (more than hundreds) theorems are given.
If so many theorems are given, we need to organize those theorems for efficientdispatching. It is a part of future work.
We mention parallelism of dispatched efficient implementations. Each of them

193

6.4. Libraries with Optimization Capabilities
4
MPS
MSP
MSS
pMPS
pMSP
pMSS
MRS
linear

3.5

speedup

3
2.5
2
1.5
1
1

1.5

2
2.5
3
the number of threads

3.5

4

Figure 6.33. Speedup of programs listed in Table 6.5 for the data set X.
exploits parallelism provided by the existing generators in Fortress, since it justsupplies newly constructed operators and functions to the seed generator held in the
GoG. Therefore, parallelism of dispatched implementations is guaranteed by thatof generators in Fortress. Figure 6.33 shows speedup of the optimized programsof Table 6.5. The figure shows good speedup of the programs. Unfortunately, thecurrent Fortress interpreter has limitations on parallelism (less than 4 times speedup
at the maximum for any programs), and thus the figure only shows the results of at
most 4 threads. This limitation will be removed in the future Fortress interpreteror compiler, and thus the programs will be able to achieve better speedup, similar
to the results on C++ implementation (see Section 6.2).
Therefore, programmers can receive the benefit that they can get the results by
the optimized efficient parallel implementation by simply writing the naive programs.
Without the library, they would suffer from writing complicated hundreds-line codefor each program.

6.4.8

Discussion

Normalization of Comprehensions for Optimization
We show, with an example, that quite a lot of nested comprehensions can be systematically transformed into the form on which current our optimization mechanism(desugaring process) works. The summary of transformations steps follows the example transformation.
Consider the next nested comprehension as the example.

194

6. Implementation of Skeletons and Optimizations
BIG MAX hPh f (y, b, w) | y ← ys, even y i |ys ← prefixes xs, ascending ys, b ← bs i

This example computes a variant of the maximum prefix sum, in which the maximum is considered only on ascending prefixes, the summation is taken only on evennumbers, and the value is replaced with f (y, b, w) instead of the number itself (y)within the summation.
The first step of the transformation is to fuse guards into their preceding generators, which is already shown in Section 6.4.5. For example, fusing guards in theexample we get the following program.
P
BIG MAX h h f (y, b, w) | y ← ys.filter (even) i |ys ← (prefixes xs).filter (ascending), b ← bs i
This transformation is applicable, when each predicate depends only on a variablein the left hand side of generators in the same comprehension.
The next step is to move depending generations to the edges. For example,
the example has the pair of depending generations y ← ys.filter (even) and ys ←(prefixes xs).filter (ascending). Since the reduction operator MAX of the example iscommutative, we can perform this transformation to get the following program.
P
BIG MAX h h f (y, b, w) | y ← ys.filter (even) i |b ← bs, ys ← (prefixes xs).filter (ascending) i
Here, ys ← (prefixes xs).filter (ascending) is moved to the edge using the commutativity.
The third step is to restructure comprehensions to extract the form. The firststage of the restructuring is to strip generators from the outer comprehension, except
for the edge generator. Applying the restructuring, we get the following program
for the example, in which the generator b ← bs is moved to a function h.
h(z) = BIG MAX h z(b)P| b ← bs ih(fn b ⇒ BIG MAX h h f (y, b, w) | y ← ys.filter (even) i |ys ← (prefixes xs).filter (ascending) i)
The second stage then strips generators and predicates from the inner comprehension. Making a new function f ′′ , we get the following result for the example.
h(z) = BIG MAX h z(b) | b ← bs ih(fn b ⇒ do f ′′ (y) = ifPeven y then f (y, b, w) else 0 end
BIG MAX h h f ′′ y | y ← ys i |ys ← (prefixes xs).filter (ascending) i
end)
Here, the argument of h includes the nested reduction of the form, and it can beprocessed by our optimization mechanism.
Here is the summary of transformations.

6.4. Libraries with Optimization Capabilities

195

Step. 1 Remove guards p x by fusing it with generators x ← g xs.

⊕[ e | x ← g xs, p x, gs ] ⇒ ⊕[ e | x ← (g xs).filter (p), gs ]Step. 2 Move depending generations to the edges. If there is depending generations
in generators of two comprehensions, move those depending generation to
the edges of comprehensions as follows.

⊕[ ⊗[ e | gs 1 ] | gs 2 ] ⇒ ⊕[ ⊗[ e | y ← fg ys, gs ′1 ] | gs ′2, ys ← fgg xs ]Here, fgg is one of GoGs with filter, and fg is the identity function or filter.
This transformation is valid if each operator of reductions is commutative
and there is no dependency of gs ′2 to ys.
Step. 3 Restructure comprehensions to extract the form. The following is a ruleused in this step.

⊕[ ⊗[ e | y ← fg ys, gs ′1 ] | gs ′2, ys ← fgg xs ]⇒ ⊕[ ⊕[ ⊗[ ⊗[ e | gs ′1 ] | y ← fg ys ] | ys ← fgg xs ] | gs ′2 ]This transformation is always valid, since it is a combination of steps usedin the usual desugaring process in Fortress. For readability, the result of
this transformation can be written as the following form.
h(⊕[ ⊗[ f ′ (y) | y ← fg ys ] | ys ← fgg xs ])where h(z) = ⊕[ z | gs ′2 ]f ′ (y) = ⊗[ e | gs ′1 ]Now, the argument of h has the desired form.

It is an interesting part of future work to implement more power full desugaring
process that can deal with the transformation.
It is worth noting that the transformation shown above has some restrictions
on target comprehensions. However, we think many practical examples satisfy the
restrictions.
Optimization of Three or More Nested Reductions
As far as we are aware, practical applications, such as nested queries on sequences,
do not actually need optimization on 3+ level deep reductions (comprehensions).
They can optimized by repeatedly applying optimizations for simply nested comprehensions (i.e., 2 level deep comprehensions), since optimization of simply nested
reductions often results in another flat reduction. We can implement such successive
optimizations on deeply nested reductions, if we let the dispatching process returna computation object to perform the flat reduction instead of immediately executing the optimized reduction. This also requires some modification on the current
desugaring process.

196

6. Implementation of Skeletons and Optimizations

Implementation in Languages Other Than Fortress
A library with the same functionality can be implemented similarly on other programming languages with subtyping and overloading, such as Java and C++. We
chose Fortress because of its features: parallelism by default, generators, and expression branching based on types. Those features enables us to make a smart and clear
library with interesting functionalities. For example, expressions branching based on
types, such as typecase, enable clear and fine control over dispatching implementations based on types. The control over priorities of overloaded functions in C++
is far more complicated and it would result in no maintainable libraries. We also
believe that the same functionality of our proposed library can be implemented infunctional languages such as Haskell [Jon02, The08] and Objective Caml [LDG+ 08].

Dynamic Dispatching and Static Dispatching
Although our current implementation uses dynamic dispatching of implementations,
we believe that the dispatching can also be done statically based on static types atcompile time. For example, we can use C++ template specialization for the static
dispatching. The static dispatching has the advantage that it has no dispatchingoverhead at runtime.
The experiment results, however, show that the overhead of the dynamic dispatching is reasonably small with respect to computation time of the reductions,and the overhead is ignorable. Also, the dynamic dispatching has another advantage that it can use runtime type information. Since it is generally finer than thestatic type information, the dynamic dispatching can apply better calculation rulesin the optimization, and this advantage would be far better than the advantage ofstatic dispatching.

Non-mathematical Properties
It is worth noting that our optimization mechanism can handle so-called “nonmathematical” properties, as long as we can make calculation theorems involving
the properties. For example, the floating point addition does not satisfy the mathematical property “associativity,” but it is approximately associative. We would
be able to develop a theory involving the “approximated associativity,” in whichcalculation theorems give efficient implementations to compute the equivalent results under the approximation. Actually, Fortress’ generators change the computational structures under the approximated associativity of reduction operators. Also,parallelization with OpenMP [CDK+ 01, CJvdP07] does the same approximation to
restructure computation of loops, and this approximation is reasonable and widely
accepted.

6.5. Related Work

6.5

197

Related WorkWe review our work in context of comprehensions, nested reduction, and rewriting.
Comprehensions and Nested Reductions
Programming using comprehension has been considered a promising approach for
concise parallelization, with a history of decades-long research [BS90,BHS+ 94,Ble96,
BG96, CK00, CK01, CKLP01, LCK06, CLJ+ 07, FRR+ 07]. The research contributed
to extracting data parallelism in aggregate computations they naturally express;
these previous approaches do not leave to user programmers detailed controls onparallel computation, for example, how data is distributed among processors or
how the systems carry out computation in parallel. Generators in Fortress allows
flexibility in this respect, enabling sophisticated tuning of programs according tovariety of their running circumstances. The previous work also studied optimization through flattening of nested comprehensions to effectively exploit a flat parallelism [BS90, LCK06] or fusion of successive operations to eliminate intermediate
data passed among them [CK01]. These efforts stayed, however, in the problem ofbalancing computation tasks of nested reductions. Our work successfully goes one
big step further to improve the complexity of computation without the hassles ofprogrammers.

Library-level Optimizations
Another area of theoretical computer science, namely term rewriting, relates toour research. Glasgow Haskell Compiler (GHC) is one existing example to employ
this technique: its RULES pragma [The08] enables users to implement library-level
optimization by rewriting based on a set of rewriting rules. For example, the short
cut fusion [GLJ93] is implemented by a set of rewriting rules in GHC. This approach,however, often suffers from inability to extract properties or surrounding contexts
for applying correct and suitable rules, and the GHC’s RULES pragma is not the
exception. Our proposed library similarly performs optimization by rewriting, butwith automatically examining specific mathematical properties of the input program
to guarantee that the rewritten output program is actually a parallel program. Our
approach therefore can be seen as an alternative to optimization by rewriting.
Broadway compiler [GL05] and the telescoping languages system [KBC+ 05] also
employ library-level optimization by rewriting. Their rewriting rules concern mathematical properties of operand values of operations, while ours mainly concern properties of operators and functions given to higher-order functions.

Skeleton Libraries and Systems
Many skeleton libraries and systems have been proposed so far, which includes
P3L [BDO+ 95,DPP97,Pel98], SCL [DFH+ 93,DGTY95], eSkel [Col04,BC05,BCHG05],

198

6. Implementation of Skeletons and Optimizations

MuesLi [Kuc02], and QUAFF [FSCL06] .
The P3L [Pel98,BDO+ 95,DPP97] system adopts the idea of separating the higher
skeletal parallel part from the lower, sequential part in parallel programs. In a P3L
program, skeletal part is written in a functional notation, while the base part is described in the C language. From these descriptions, the P3L compiler generates a C
code that calls MPI library functions. The system supports data parallel skeletonsand communication skeletons for distributed lists. It also supports control parallel skeletons. The selection of the set of data parallel skeletons is similar to ourframework.
In Darlington’s framework SCL [DFH+ 93, DGTY95], the user writes a parallel
program in two-layer structure: higher skeleton level and lower base language level.
The user writes the higher level of the program with skeletons, abstracting its parallel behavior using the SCL (Structured Coordination Language). Its syntax is somekind of functional notation. The lower sequential part of the program is describedin the base language. The system supports data parallel skeletons and communication skeletons for distributed arrays (one or two dimensional), and control parallelskeletons.
eSkel [Col04, BC05, BCHG05] is a library of C functions, also implemented on
top of MPI. The latest version of eSkel supports control parallel skeletons, puttingemphasis on addressing the issues of nesting of skeletons and interaction between
parallel activities.
Muesli, developed by Kuchen [Kuc02], is a C++ library that works using MPI.
It supports data parallel skeletons for distributed list and matrix, and task parallelskeletons. The system has two tier model: a parallel computation consists of asequence of independent task parallel computations where each computation may
nest task parallel skeletons arbitrarily, and an atomic (i.e. non-nested) task parallel
computation can contain data parallelism. However, nesting constructors for thecorresponding skeletons to generate a process topology is sometimes hard for theuser because of low abstraction of task parallel skeletons.
QUAFF [FSCL06] is a skeleton-based parallel programming library on C++.
It supports three task parallel skeletons. Its main originality is to rely on C++
template meta-programming techniques to achieve high efficiency. In particular, byperforming most of skeleton instantiation and optimization at compile-time, QUAFF
can keep the overhead very small.
APL [Ive62,FI73,Ber93] is a pioneer language that supports operators to manipulate arrays, namely array operators. APL’s array operators supports element-wisecomputations, reductions to collapse arrays with binary operators, and manipulation of the layout of arrays such as shift and rotate. More complex computations
on arrays can be made by composing the array operators. The idea of APL is thesame as skeletal parallel programming, in the sense that both provide users with aset of basic patterns of array computations, and let users make sophisticated computations by compositions of these basic patterns. Also, both array operators andskeletons conceal complicated parallelism from users.

Chapter 7
Conclusion
7.1

Summary of the Thesis

This thesis has studied homomorphism-based structured parallel programming (alsoknown as skeletal parallel programming), in which the skeletons to organize parallel
programs have been designed based on homomorphisms of algebras of data structures. Structured by homomorphisms, the designed skeletons have good composability and good optimizability.
In the first part of the thesis, we have studied the homomorphism-based design of parallel skeletons for lists, two-dimensional arrays, and trees. Structured byhomomorphisms, the designed skeletons have good composability and good optimizability. Especially, we have successfully designed skeletons for two-dimensional
arrays, which had remained as a challenging problem. We have used the abide-tree
algebra to represent two-dimensional arrays, which has nice freedom for parallelism
owing to the abide property of constructors. The abide-tree algebra can be seen as
an extension of the monoid algebra. We have shown that the designed skeletons arewell composable with each other to build parallel programs for various problems.
In the second part, we have studied optimization of skeleton programs to solve
the inefficiency problem of the compositional-style programming of skeletons. Basedon the nice fusion laws of homomorphisms, we have successfully developed domainindependent fusion optimization for the skeletons, in which consecutive skeletonsare fused into one skeleton to eliminate the redundant intermediate data structures
between them. We have shown a derivation of non-trivial efficient programs forthe maximum rectangle sum problem from naively-composed skeleton programs,
as well as a general strategy for the derivation. The result shows that we cansystematically develop non-trivial efficient algorithms from naive compositions of ourskeletons. Also, we have proposed studied domain-specific optimizations for skeleton
programs, to solve problems of the domain-independent fusion optimizations. We
have concentrated on optimization of skeleton programs for computation involving
neighbor elements, which can be categorized into two types: that involving a finite
number of neighbor elements, such as filtering of sequences and images, the finite

200

7. Conclusion

difference method, and some matrix-vector operations; and that involving an infinite
number of neighbor elements, such as queries of interesting segments on lists andrectangles (sub-arrays) on two-dimensional arrays. We have developed domainspecific fusion rules for the former type, proposing a new strategy for developing
domain-specific fusion optimization of skeleton programs. Then, for the latter typethat have been formalized as nested reductions, we have developed shortcut theorems
to provide efficient algorithms to nested reductions by fusion. We have succeeded indeveloping the domain-specific fusions and the optimization theorems, owing to thesolid theories of carefully designed homomorphism-based skeletons. Also, we havesucceeded in widening the application area of our skeleton programming.
In the last part of the thesis, we have reported implementations of designed
skeletons and optimization mechanisms. We have shown implementations of skeletons for distributed parallel machines, in which the parallelism of homomorphism
guarantees their parallelism. We have shown small systems for domain-independentand domain-specific fusions, which have been implemented as source-to-source translators. Experiment results show the success of the implementations and the basetheories. We have also proposed the general design of libraries with optimization
capabilities based on optimization theorems in our theories. The design has been
demonstrated with the implementation of the GoG library for nested reductions
in Fortress, in which the useful comprehension notation is linked to our skeleton
theories. The implementation of the optimizing libraries is lightweight in the sensethat it does not need deep analysis of program codes. The key point is the factthat application conditions of theorems are described with parameters’ possessionof mathematical properties, which is due to the uniform structure of skeletons basedon homomorphisms, i.e., higher-order functions. The implemented libraries can beseen as active dictionaries of algorithms, in the sense that efficient implementations
are indexed in the dictionaries by naive programs.

7.2

Future Work

Although we have shown a strategy to make efficient parallel algorithms from naive
compositions of skeletons, there is still a demand for methodologies to construct
the initial naive compositions. Especially, there is a demand for a strong methodology to derive operators with the abide property, to derive naive parallel programsfor two-dimensional arrays. For skeletal parallel programs on one-dimensional data
structures, some techniques have been proposed to derive skeleton programs (associative operators). The key of one technique is the use of the associativity offunction compositions as the seed of associative operators. We might be able toapply the key idea to derivation of abide operators. We have found two pairs ofsuch seed operations: the pair of function composition and function tupling, andthe pair of matrix multiplication and tensor product. Unfortunately, neither of seedpairs results in useful operator pairs, since one of the derived pair operators builds

7.2. Future Work201

big structures, in other words, it simply delays computations of its direction. Thus,
investigation of good seed pairs is one part of future work.
Extension of parallel skeletons to multi-dimensional data structures is also one
direction of future work. We have not found many application problems that essentially need multi-dimensional data structures, though.
Automatic discovery of mathematical properties of program objects is very usefulfor automatic optimization with calculation theorems. Also, a technique to solve
priority problems in applying calculation theorems is still an open problem [PP96],
since a greedy selection of a sequence of theorems does not necessarily result in thebest implementation. We have avoided such priority problem by implementing only
big-step theorems. However, the problem would arise again when the number ofbig-steps becomes large. Study of these techniques is one direction of future work.
It is also interesting future work to study the integration of data-parallel skeletonsand task-parallel skeletons. The computation provided by task-parallel skeletons distributes tasks among processors, while the computation by data-parallel skeletons
distributes data structures among processors, and carries out almost the same process on each processor. This thesis have studied data-parallel skeletons structured
by homomorphisms on data structures, since the center of their computation is thedata structure. However, it is an open problem what should be used to structuretask-parallel skeletons. It is also an open problem whether we can integrate the theory of structured task-parallel skeletons into the theory of structured data-parallelskeletons.
Although we have shown that some matrix operations can be described with
our skeletons, some matrix operations are not so easily written with the designed
skeletons. Also, conditions for optimizing these operations are often described withproperties of input matrices, while the optimization of our skeletons have conditions
on operators (functions). Therefore, we might have to take another approach tostructuring and optimizing those operations. Some results are found in a technicalreport [EHK+ 09].

Bibliography[ACH+ 08] Eric Allen, David Chase, Joe Hallett, Victor Luchangco, Jan-Willem
Maessen, Sukyoung Ryu, Guy L. Steele Jr., and Sam Tobin-Hochstadt.
The Fortress language specification version 1.0. http://research.sun.
com/projects/plrg/fortress.pdf, 2008.
[ADKP89] Karl R. Abrahamson, N. Dadoun, David G. Kirkpatrick, and Teresa M.
Przytycka. A simple parallel tree contraction algorithm. Journal of
Algorithms, 10(2):287–302, June 1989.
[BBC+ 06] Anders Berglund, Scott Boag, Don Chamberlin, Mary F. Fernández,
Michael Kay, Jonathan Robie, and Jérôme Siméon, editors. XML Path
Language (XPath) 2.0. W3C Candidate Recommendation, June 2006.
available from http://www.w3.org/TR/xpath20/.
[BC05]

Anne Benoit and Murray Cole. Two fundamental concepts in skeletal
parallel programming. In International Conference on Computational
Science (2), volume 3515 of Lecture Notes in Computer Science, pages
764–771. Springer-Verlag, 2005.

[BCHG05] Anne Benoit, Murray Cole, J. Hillston, and S. Gilmore. Flexible skeletal
programming with eskel. In Proceedings of 11th International Euro-Par
Conference (Euro-Par’05), volume 3648 of Lecture Notes in Computer
Science, pages 761–770. Springer-Verlag, 2005.
[BdM96]

Richard S. Bird and Oege de Moor. Algebra of Programming. International Series in Computer Science. Prentice Hall, September 1996.

[BDO+ 95] Bruno Bacci, Marco Danelutto, Salvatore Orlando, Susanna Pelagatti,
and Marco Vanneschi. P3L: A Structured High Level Programming Language and its Structured Support. Concurrency Practice and Experience,7(3):225–255, 1995.
[Ben84a]

Jon Bentley. Programming Pearls: Algorithm Design Techniques. Communications of the ACM, 27(9):865–873, 1984.

[Ben84b]

Jon Bentley. Programming Pearls: Perspective on Performance. Communications of the ACM, 27(11):1087–1092, 1984.

204[Ber93]

R. Bernecky. The role of APL and J in high-performance computation.
APL Ouote Ouad, 24(1):17–32, 1993.

[BG96]Guy E. Blelloch and John Greiner. A provable time and space efficient
implementation of nesl. In ICFP ’96: Proceedings of the first ACM
SIGPLAN international conference on Functional programming, pages
213–225, New York, NY, USA, 1996. ACM.

[BGH+ 06] Ganesh Bikshandi, Jia Guo, Daniel Hoeflinger, Gheorghe Almasi,
Basilio B. Fraguela, Maria J. Garzaran, David Padua, and Christophvon Praun. Programming for Parallelism and Locality with Hierarchically Tiled Arrays. In Proceedings of 11th ACM SIGPLAN Symposiumon Principles and Practice of Parallel Programming (PPoPP’06), pages
48–57, New York, NY, USA, 2006. ACM Press.
[BHS+ 94] Guy E. Blelloch, Jonathan C. Hardwick, Jay Sipelstein, Marco Zagha, and Siddhartha Chatterjee. Implementation of a portable nested
data-parallel language. Journal of Parallel and Distributed Computing,21(1):4–14, 1994.
[Bir87]

Richard S. Bird. An introduction to the theory of lists. In Proceedings of
the NATO Advanced Study Institute on Logic of programming and calculi
of discrete design, pages 5–42, New York, NY, USA, 1987. SpringerVerlag New York, Inc.

[Bir88]Richard S. Bird. Lectures on constructive functional programming. Technical Report Technical Monograph PRG-69, Oxford University Computing Laboratory, 1988.

[Bir98]Richard S. Bird. Introduction to Functional Programming using Haskell.
Prentice Hall Series in Computer Science. Prentice Hall, 2nd edition,
April 1998.

[Bir01]

Richard S. Bird. Functional pearls maximum marking problems. Journalof Functional Programming, 11(4):411–424, 2001.

[Ble90]Guy E. Blelloch. Prefix sums and their applications. Technical Report
CMU-CS-90-190, School of Computer Science, Carnegie Mellon University, November 1990.

[Ble96]

Guy E. Blelloch. Programming parallel algorithms. Communication
ACM, 39(3):85–97, 1996.

[BS90]Guy E. Blelloch and Gary W. Sabot. Compiling collection-oriented languages onto massively parallel computers. Journal of Parallel and Distributed Computing, 8(2):119–134, 1990.

Bibliography

205[BSWB02] David A. Bader, Sukanya Sreshta, and Nina R. Weisse-Bernstein. Evaluating arithmetic expressions using tree contraction: A fast and scalable parallel implementation for symmetric multiprocessors (SMPs) (extended abstract). In Sartaj Sahni, Viktor K. Prasanna, and Uday Shukla,
editors, High Performance Computing — HiPC 2002, 9th International
Conference, Bangalore, India, December 18–21, 2002, Proceedings, volume 2552 of Lecture Notes in Computer Science, pages 63–78. Springer,
2002.
[CDK+ 01] Robit Chandra, Leonardo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, and Ramesh Menon. Parallel programming in OpenMP. Morgan
Kaufmann Publishers Inc., San Francisco, CA, USA, 2001.
[CJvdP07] Barbara Chapman, Gabriele Jost, and Ruud van der Pas. Using
OpenMP: Portable Shared Memory Parallel Programming (Scientific andEngineering Computation). The MIT Press, 2007.
[CK00]

Manuel M. T. Chakravarty and Gabriele Keller. More types for nested
data parallel programming. In ICFP ’00: Proceedings of the fifth ACM
SIGPLAN international conference on Functional programming, pages
94–105, New York, NY, USA, 2000. ACM.

[CK01]

Manuel M. T. Chakravarty and Gabriele Keller. Functional array fusion.
In ICFP ’01: Proceedings of the sixth ACM SIGPLAN international
conference on Functional programming, pages 205–216, New York, NY,
USA, 2001. ACM.

[CKLP01] Manuel M. T. Chakravarty, Gabriele Keller, Roman Lechtchinsky, andW. Pfannenstiel. Nepal - nested data parallelism in haskell. In Euro-Par’01: Proceedings of the 7th International Euro-Par Conference Manchester on Parallel Processing, pages 524–534, London, UK, 2001. SpringerVerlag.
[CLJ+ 07] Manuel M. T. Chakravarty, Roman Leshchinskiy, Simon Peyton Jones,
Gabriele Keller, and Simon Marlow. Data parallel haskell: a status
report. In DAMP ’07: Proceedings of the 2007 workshop on Declarative
aspects of multicore programming, pages 10–18, New York, NY, USA,2007. ACM.
[Col89]Murray Cole. Algorithmic Skeletons : A Structured Approach to theManagement of Parallel Computation. Research Monographs in Paralleland Distributed Computing, Pitman, London, 1989.

[Col95]

Murray Cole. Parallel programming with list homomorphisms. Parallel
Processing Letters, 5(2):191–203, June 1995.

206[Col02]

Murray
Cole.
eSkel
Home
http://homepages.inf.ed.ac.uk/mic/eSkel/, 2002.

Page.

[Col04]

Murray Cole. Bringing skeletons out of the closet: a pragmatic manifesto
for skeletal parallel programming. Parallel Comput., 30(3):389–406, 2004.

[CTH98]

Wei-Ngan Chin, Akihiko Takano, and Zhenjiang Hu. Parallelization via
context preservation. In Proceedings of the 1998 International Conference
on Computer Languages, ICCL ’98, May 14–16, 1998, Chicago, IL, USA,pages 153–162. IEEE Computer Society, 1998.

[CV88]

Richard Cole and Uzi Vishkin. The accelerated centroid decomposition
technique for optimal parallel tree evaluation in logarithmic time. Algorithmica, 3:329–346, March 1988.

[DFH+ 93] John Darlington, A. J. Field, Peter G. Harrison, Paul H. J. Kelly,
D. W. N. Sharp, and Q. Wu. Parallel programming using skeleton functions. In PARLE ’93: Proceedings of the 5th International PARLE Conference on Parallel Architectures and Languages Europe, pages 146–160,
London, UK, 1993. Springer-Verlag.
[DGTY95] John Darlington, Yi-ke Guo, Hing Wing To, and Jin Yang. Parallelskeletons for structured composition. SIGPLAN Not., 30(8):19–28, 1995.
[DH98]

Krzysztof Diks and Torben Hagerup. More general parallel tree contraction: Register allocation and broadcasting in a tree. TheoreticalComputer Science, 203(1):3–29, August 1998.

[DPP97]

M. Danelutto, F. Pasqualetti, and S. Pelagatti. Skeletons for data parallelism in P3L. In Proceedings of 3rd International Euro-Par Conference(Euro-Par’97), volume 1300 of Lecture Notes in Computer Science, pages
619–628. Springer-Verlag, 1997.

[EGJK04] Erik Elmroth, Fred Gustavson, Isak Jonsson, and Bo Kagstroom. Recursive Blocked Algorithms and Hybrid Data Structures for Dense Matrix
Library Software. SIAM Review, 46(1):3–45, 2004.
[EHK+ 09] Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Kiminori Matsuzaki,
and Masato Takeichi. A generative matrix library in fortress relieves
programmers’ headache! In Proceedings of the 25th JSSST domestic
conference, 2009.
[FI73]

Adin D. Falkoff and Kenneth E. Iverson. The design of APL. IBM
Journal of Research and Development, 17(4):324–334, 1973.

[For08]

Project Fortress. The reference interpreter for the Fortress language.
http://projectfortress.sun.com/Projects/Community, 2008.

Bibliography[FPS01]

207

Paul F. Fischer, Franco P. Preparata, and John E. Savage. Generalized
scans and tridiagonal systems. Theoretical Computer Science, 255(1–2):423–436, 2001.

[FRR+ 07] Matthew Fluet, Mike Rainey, John Reppy, Adam Shaw, and Yingqi Xiao.
Manticore: a heterogeneous parallel language. In DAMP ’07: Proceedingsof the 2007 workshop on Declarative aspects of multicore programming,
pages 37–44, New York, NY, USA, 2007. ACM.
[FSCL06] J. Falcou, J. Sérot, T. Chateau, and J. T. Lapresté. Quaff: efficient c++design for parallel skeletons. Parallel Comput., 32(7):604–615, 2006.
[FW03]

Jeremy D. Frens and David S. Wise. QR Factorization with MortonOrdered Quadtree Matrices for Memory Re-use and Parallelism. In Proceedings of 4th ACM SIGPLAN Symposium on Principles and Practiceof Parallel Programming (PPoPP’03), pages 144–154, 2003.

[GCS94]

Jeremy Gibbons, Wentong Cai, and David B. Skillicorn. Efficient Parallel
Algorithms for Tree Accumulations. Science of Computer Programming,
23(1):1–18, 1994.

[GJSB05] James Gosling, Bill Joy, Guy Steele, and Gilad Bracha. Java(TM)Language Specification, The (3rd Edition) (Java (Addison-Wesley)).
Addison-Wesley Professional, 2005.
[GL05]

Samuel Zev Guyer and Calvin Lin. Broadway: A compiler for exploiting
the domain-specific semantics of software libraries. Proceedings of theIEEE, 93(2):342–357, 2005.

[GLJ93]

Andrew Gill, John Launchbury, and Simon L. Peyton Jones. A short cut
to deforestation. In FPCA ’93: Proceedings of the conference on Functional programming languages and computer architecture, pages 223–232,
New York, NY, USA, 1993. ACM.

[GLS99]William Gropp, Ewing Lusk, and Anthony Skjellum. Using MPI (2nded.): portable parallel programming with the message-passing interface.
MIT Press, Cambridge, MA, USA, 1999.

[Gor96]

Sergei Gorlatch. Systematic efficient parallelization of scan and otherlist homomorphisms. In Luc Bougé, Pierre Fraigniaud, Anne Mignotte,and Yves Robert, editors, Euro-Par ’96 Parallel Processing, Second International Euro-Par Conference, Lyon, France, August 26–29, 1996,
Proceedings, Volume II, volume 1124 of Lecture Notes in Computer Science, pages 401–408. Springer, 1996.

208[Gor97]Sergei Gorlatch. Optimizing compositions of scans and reductions inparallel program derivation. Technical Report MPI–9711, Universität
Passau, 1997.

[GS06]

Clemens Grelck and Sven-Bodo Scholz. Merging compositions of arrayskeletons in SaC. Parallel Computing, 32(7–8):507–522, 2006.

[GW06]

Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing(3rd Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2006.

[GWL99] Sergei Gorlatch, Christoph Wedler, and Christian Lengauer. Optimization rules for programming with collective operations. In 13th International Parallel Processing Symposium / 10th Symposium on Parallel andDistributed Processing (IPPS / SPDP ’99), 12-16 April 1999, San Juan,
Puerto Rico, Proceedings, pages 492–499. IEEE Computer Society, 1999.
[HIT97]

Zhenjiang Hu, Hideya Iwasaki, and Masato Takeichi. Formal Derivationof Efficient Parallel Programs by Construction of List Homomorphisms.
ACM Transactions on Programming Langauges and Systems, 19(3):444–
461, 1997.

[HIT02]

Zhenjiang Hu, Hideya Iwasaki, and Masato Takeichi. An Accumulative
Parallel Skeleton for All. In Proceedings of 11st European Symposium on
Programming (ESOP 2002), LNCS 2305, pages 83–97. Springer-Verlag,
April 2002.

[HTC98]

Zhenjiang Hu, Masato Takeichi, and Wei-Ngan Chin. Parallelizationin calculational forms. In POPL ’98, Proceedings of the 25th ACM
SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 19–21, 1998, San Diego, CA, USA, pages 316–328.
ACM Press, 1998.

[HTI99]Zhenjiang Hu, Masato Takeichi, and Hideya Iwasaki. Diffusion: Calculating efficient parallel programs. In Olivier Danvy, editor, Proceedings of the 1999 ACM SIGPLAN Workshop on Partial Evaluation and
Semantics-Based Program Manipulation, San Antonio, Texas, January
22–23, 1999, pages 85–94. University of Aarhus, 1999. Technical report
BRICS-NS-99-1.

[Ive62]Kenneth E. Iverson. A Programming Language. John Wiley and Sons,
1962.

[Jai89]Anil K. Jain. Fundamentals of Digital Image Processing. Prentice Hall,
1989.

[Jeu93]

Johan Theodoor Jeuring. Theories for Algorithm Calculation. PhD thesis, Faculty of Science, Utrecht University, 1993.

Bibliography[Jon02]

209

Simon Peyton Jones. Haskell 98 language and libraries: The revised
report. PDF, September 2002.

[KBC+ 05] Ken Kennedy, Bradley Broom, Arun Chauhan, Rob Fowler, John
Garvin, Charles Koelbel, Cheryl McCosh, and John Mellor-Crummey.
Telescoping languages: A system for automatic generation of domain
languages. Proceedings of the IEEE, 93(2):387–408, Feb. 2005.
[Kuc02]

Herbert Kuchen. A Skeleton Library. In Proceedings of 8th International
Euro-Par Conference (Euro-Par’02), volume 2400 of Lecture Notes in
Computer Science, pages 620–629. Springer-Verlag, 2002.

[LCK06]

Roman Leshchinskiy, Manuel M. T. Chakravarty, and Gabriele Keller.
Higher order flattening. In Vassil N. Alexandrov, G. Dick van Albada,Peter M. A. Sloot, and Jack Dongarra, editors, Computational Science- ICCS 2006, 6th International Conference, Reading, UK, May 28-31,
2006, Proceedings, Part II, volume 3992 of Lecture Notes in Computer
Science, pages 920–928. Springer, 2006.

[LDG+ 08] Xavier Leroy, Damien Doligez, Jacques Garrigue, Didier Remy, andJerome Vouillon. The Objective Caml system release 3.11. http://caml.inria.fr/, November 2008.
[LM88]Charles E. Leiserson and Bruce M. Maggs. Communication-efficient parallel algorithms for distributed random-access machines. Algorithmica,
3:53–77, 1988.

[Mat07]

Kiminori Matsuzaki. Parallel Programming with Tree Skeletons. PhD
thesis, Graduate School of Information Science and Technology, University of Tokyo, 2007.

[Mee88]Lambert Meertens. First steps towards the theory of rose trees. CWI,
Amsterdam; IFIP Working Group 2.1 working paper 592 ROM-25, 1988.

[MFP91]Erik Meijer, Maarten M. Fokkinga, and Ross Paterson. Functional programming with bananas, lenses, envelopes and barbed wire. In Proceedings of the 5th ACM Conference on Functional Programming Languages
and Computer Architecture, pages 124–144, London, UK, 1991. SpringerVerlag.

[MIEH06] Kiminori Matsuzaki, Hideya Iwasaki, Kento Emoto, and Zhenjiang Hu.
A library of constructive skeletons for sequential style of parallel programming. In InfoScale ’06: Proceedings of the 1st international conference on Scalable information systems, volume 152 of ACM International
Conference Proceeding Series, page 13. ACM Press, 2006.

210[Mil94]

Richard Miller. Two Approaches to Architecture-Independent Parallel
Computation. PhD thesis, Computing Laboratory, Oxford University,
1994.

[MKI+ 04] Kiminori Matsuzaki, Kazuhiko Kakehi, Hideya Iwasaki, Zhenjiang Hu,
and Yoshiki Akashi. A Fusion-Embedded Skeleton Library. In Proceedings of 10th International Euro-Par Conference (Euro-Par’04), volume3149 of Lecture Notes in Computer Science, pages 644–653. SpringerVerlag, 2004.
[MR85]

Gary L. Miller and John H. Reif. Parallel tree contraction and its application. In 26th Annual Symposium on Foundations of Computer Science,
21–23 October 1985, Portland, Oregon, USA, pages 478–489. IEEE Computer Society, 1985.

[MW97]

Ernst W. Mayr and Ralph Werchner. Optimal tree contraction andterm matching on the hypercube and related networks. Algorithmica,
18(3):445–460, July 1997.

[OHIT97] Yoshiyuki Onoue, Zhenjiang Hu, Hideya Iwasaki, and Masato Takeichi.
A calculational fusion system hylo. In Proceedings of the IFIP TC 2 WG
2.1 international workshop on Algorithmic languages and calculi, pages
76–106, London, UK, UK, 1997. Chapman & Hall, Ltd.
[Pel98]Susanna Pelagatti. Structured development of parallel programs. Taylor& Francis, Inc., Bristol, PA, USA, 1998.

[PP96]

Alberto Pettorossi and Maurizio Proietti. Rules and strategies fortransforming functional and logic programs. ACM Computing Surveys,
28(2):360–414, 1996.

[RG02]

Fethi A. Rabhi and Sergei Gorlatch, editors. Patterns and Skeletons for
Parallel and Distributed Computing. Springer-Verlag, 2002.

[RT94]

John H. Reif and Stephen R. Tate. Dynamic parallel tree contraction (extended abstract). In Lawrence Snyder and Charles E. Leiserson, editors,
SPAA ’94: Proceedings of the 6th Annual ACM Symposium on Parallel
Algorithms and Architectures, June 27–29, 1994, Cape May, New Jersey,
USA, pages 114–121. ACM Press, 1994.

[Rus06]

John C. Russ. The Image Processing Handbook, Fifth Edition (Image
Processing Handbook). CRC Press, Inc., Boca Raton, FL, USA, 2006.

[SG02]

Jocelyn Serot and Dominique Ginhac. Skeletons for Parallel Image Processing: an Overview of the SKIPPER Project. Parallel Computing,28(12):1685–1708, 2002.

Bibliography[SHT01]211

Isao Sasano, Zhenjiang Hu, and Masato Takeichi. Generation of efficient programs for solving maximum multi-marking problems. In SAIG
2001: Proceedings of the Second International Workshop on Semantics,
Applications, and Implementation of Program Generation, pages 72–91,
London, UK, 2001. Springer-Verlag.

[SHTO00] Isao Sasano, Zhenjiang Hu, Masato Takeichi, and Mizuhito Ogawa.
Make it practical: a generic linear-time algorithm for solving maximumweightsum problems. In ICFP ’00: Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, pages 137–
149, New York, NY, USA, 2000. ACM.
[Ski94]David B. Skillicorn. Foundations of Parallel Programming, volume 6of Cambridge International Series on Parallel Computation. Cambridge
University Press, 1994.

[Ski96]David B. Skillicorn. Parallel implementation of tree skeletons. Journalof Parallel and Distbributed Computing, 39(2):115–125, December 1996.

[Ski97]David B. Skillicorn. Structured parallel computation in structured documents. Journal of Univarsal Computer Science, 3(1):42–68, January
1997.

[SO98]

Marc Snir and Steve Otto. MPI-The Complete Reference: The MPI
Core. MIT Press, Cambridge, MA, USA, 1998.

[Sto73]Harold S. Stone. An efficient parallel algorithm for the solution of atridiagonal linear system of equations. Journal of the ACM, 20(1):27–
38, 1973.

[Sve02]

Josef Svenningsson. Shortcut fusion for accumulating parameters &zip-like functions. In ICFP ’02: Proceedings of the seventh ACM SIGPLAN international conference on Functional programming, pages 124–
132, New York, NY, USA, 2002. ACM.

[Tak02]

Tadao Takaoka. Efficient Algorithms for the Maximum Subarray Problem by Distance Matrix Multiplication. In Proceedings of Computing:
The Australasian Theory Symposium (CATS’02), pages 189–198, 2002.

[The08]The GHC Team. The glorious glasgow haskell compilation system user’s
guide, version 6.10.1. http://www.haskell.org/ghc/, 2008.

[TM95]

Akihiko Takano and Erik Meijer. Shortcut deforestation in calculational
form. In FPCA ’95: Proceedings of the seventh international conference
on Functional programming languages and computer architecture, pages
306–313, New York, NY, USA, 1995. ACM.

212
[Wad88]

Philip Wadler. Deforestation: Transforming programs to eliminate trees.
In Harald Ganzinger, editor, ESOP ’88, 2nd European Symposium on
Programming, Nancy, France, March 21-24, 1988, Proceedings, volume
300 of Lecture Notes in Computer Science, pages 344–358. Springer, 1988.

[Wis84]David S. Wise. Representing Matrices as Quadtrees for Parallel Processors. Information Processing Letters, 20(4):195–199, 1984.

[Wis99]David S. Wise. Undulant Block Elimination and Integer-Preserving Matrix Inversion. Science of Computer Programming, 22(1):29–85, 1999.

[WL98]

Christoph Wedler and Christian Lengauer. On linear list recursion inparallel. Acta Informatica, 35(10):875–909, October 1998.

[XKH04]Dana N. Xu, Siau-Cheng Khoo, and Zhenjiang Hu. PType system: Afeatherweight parallelizability detector. In Wei-Ngan Chin, editor, Programming Languages and Systems: Second Asian Symposium, APLAS
2004, Taipei, Taiwan, November 4–6, 2004. Proceedings, volume 3302 of
Lecture Notes in Computer Science, pages 197–212. Springer, 2004.

[Zan92]

Hans Zantema. Longest segment problems. Science of Computer Programming, 18(1):39–66, 1992.

[Zha02]

Haiyan Zhao. A Compositional Approach to Mining Optimal Ranges.
PhD thesis, Department of Information Engineering, University ofTokyo, 2002.

Appendix A
Auxiliary Rules for Skeletons on
Two-dimensional Arrays
Rule I
map f (zipwith(⊕) x y) = zipwith(⊕′ ) (map f x) (map f y)
⇐ ∀a, b f (a ⊕ b) = f a ⊕′ f b
Proof: It is proved by induction on the structure of abide trees.
===

map f (zipwith(⊕) (x y) (u v)){ similar to −◦}zipwith(⊕′ ) (map f (x y)) (map f (u
−
◦

=

−
◦

=

v))

−
◦

=

map f (zipwith(⊕) (x−◦ y) (u−◦ v)){ def. of zipwith, map }map f (zipwith(⊕) x u)−◦ map f (zipwith(⊕) y v)
{ hypo. of induction }zipwith(⊕′ ) (map f x) (map f u)−◦ zipwith(⊕′ ) (map f y) (map f v)
{ def. of zipwith, map }zipwith(⊕′ ) (map f (x−◦ y)) (map f (u−
◦ v))
−
◦

=

map f (zipwith(⊕) |a| |b|){ def. of zipwith, map }|f (a ⊕ b)|{ hypo. }|f a ⊕′ f b|
{ def. of zipwith, map }zipwith(⊕′ ) (map f |a|) (map f |b|)

Rule IImap (reduce(⊕, ⊗)) (zipwith(−◦) x y)= zipwith(⊕) (map (reduce(⊕, ⊗)) x) (map (reduce(⊕, ⊗)) y)214
Proof: Rule I and the following calculation with f = reduce(⊕, ⊗), ⊕ = −◦, ⊕′ = ⊕ .
reduce(⊕, ⊗) (a−◦ b) = reduce(⊗, ⊕) a ⊕ reduce(⊗, ⊕) b

Rule III
map f (gemm(⊕, ⊗) x y) = gemm(⊕′ , ⊗′ ) (map f x) (map f y)
⇐ ∀a, b f (a ⊕ b) = f a ⊕′ f b, f (a ⊗ b) = f a ⊗′ f b
Proof: It is proved by induction on the structure of abide trees.
===

map f (gemm(⊕, ⊗) |a| |b|){ def. of gemm, map }|f (a ⊗ b)|{ hypo. }|f a ⊗′ f b|
{ def. of gemm, map }gemm(⊕′ , ⊗′ ) (map f |a|) (map f |b|)

map f (gemm(⊕, ⊗) (x−◦ y) z){ def. of gemm, map }map f (gemm(⊕, ⊗) x z)−◦ map f (gemm(⊕, ⊗) y z)=
{ hypo. of induction }gemm(⊕′ , ⊗′ ) (map f x) (map f z)−◦ gemm(⊕′ , ⊗′ ) (map f y)) (map f z)=
{ def. of gemm, map }gemm(⊕′ , ⊗′ ) (map f (x−◦ y)) (map f z)=

map f (gemm(⊕, ⊗) x (y z)){ similar to above }gemm(⊕′ , ⊗′ ) (map f x) (map f (y
−
◦

z))

−
◦

=

◦ v))
map f (gemm(⊕, ⊗) (x y) (u−{ def. of gemm, map }map f (zipwith(⊕) (gemm(⊕, ⊗) x u) (gemm(⊕, ⊗) y v)){I}zipwith(⊕′ )(map f (gemm(⊕, ⊗) x u)) (map f (gemm(⊕, ⊗) y v)){ hypo. of induction }zipwith(⊕′ )(gemm(⊕′ , ⊗′ ) (map f x) (map f u))(gemm(⊕′ , ⊗′ ) (map f y) (map f v)){ def. of gemm, map }gemm(⊕′ , ⊗′ ) (map f (x−◦ y)) (map f z)
−
◦

=
=
=

=

Rule IV
map f (map(⊕x) y) = map (⊗′ (f x))(map f y)
⇐ ∀a, b f (a ⊕ b) = f a ⊕′ f b215Bibliography
Proof: It is proved by induction on the structure of abide trees.
===

map f (map(⊕x) |a|){ def. of map }|f (x ⊕ a)|{ hypo. }|f x ⊕′ f a)|{ def. of map }map (⊗′ (f x))(map f y)−◦

map f (map(⊕x) (y z)){ def. of map }map f (map(⊕x) y) map f (map(⊕x) z){ hypo. of induction }map (⊗′ (f x))(map f y) map (⊗′ (f x))(map f z)
{ def. of map }map (⊗′ (f x))(map f (y z))
−
◦

=

−
◦

=

−
◦

=

The inductive case for −◦ is proved similarly.
The following is an instance of this rule:
map sum (zipwith (−◦) a b) = zipwith(+) (map sum a) (map sum b)

Rule Vmap f (right ′ x) = right ′ (map (map f ) x)
Proof:======

map f ◦ right ′{ def. of right ′ }map f ◦ the ◦right{ def. of right }map f ◦ the ◦ reduce(−◦, ≫) ◦ map | · |{ def. of the, map }the ◦ map (map f ) ◦ reduce(−◦, ≫) ◦ map | · |{ VI }the ◦ reduce(−◦, ≫) ◦ map (map (map f )) map | · |{ def. of | · |, map }the ◦ reduce(−◦, ≫) ◦ map | · | ◦ map (map f )){ def. of right ′ }right ′ ◦ map (map f )This rule for top ′ holds similarly.

Rule VI−◦

map f ◦ reduce(⊕, ⊗) = reduce(⊕, ⊗) ◦ map (map f )⇐ ⊕, ⊗ ∈ {−◦, , ≪, ≫}216
Proof:
map f (reduce(⊕, ⊗) |a|){ def. of reduce }map f a={ def. of reduce }reduce(⊕, ⊗) | map f a|=
{ def. of map }reduce(⊕, ⊗) (map (map f ) |a|)=map f (reduce(⊕, ⊗) (x y)){ def. of reduce }map f (reduce(⊕, ⊗) x ⊗ reduce(⊕, ⊗)y){ below }map f (reduce(⊕, ⊗) x) ⊗ map f (reduce(⊕, ⊗)y){ hypo. of induction }reduce(⊕, ⊗) (map (map f ) x) ⊗ reduce(⊕, ⊗) (map (map f ) y)
{ def. of map, reduce }reduce(⊕, ⊗) (map (map f ) (x y))
−
◦

=
=
=

−
◦

=

The inductive case for −◦ is proved similarly.

−◦

map f (x ⊕ y) = map f x ⊕ map f y ⇐ ⊕ ∈ {−◦, , ≪, ≫}Proof:−◦−◦

map f (x y) = map f x map f y
map f (x−◦ y) = map f x−
◦ map f y
map f (x ≫ y) = map f y = map f x ≫ map f y
map f (x ≪ y) = map f x = map f x ≪ map f y

Rule VII
map f (zipwith4 g x u w a)
= zipwith4 g ′ (map f1 x) (map f2 u) (map f3 w) (map f4 a)⇐ f (g x u w a) = g ′ (f1 x) (f2 u) (f3 w) (f4 a)

Proof: It is proved by induction on the structure of abide trees.
===

map f (zipwith4 g |a| |b| |c| |d|){ def. of zipwith, map }|f (g a b c d)|{ hypo. }|g ′ (f1 a) (f2 b) (f3 c) (f4 d)|{ def. of zipwith, map }zipwith4 g ′ (map f1 |a|) (map f2 |b|) (map f3 |c|) (map f4 |d|)
−
◦

−
◦

−
◦

−
◦

map f (zipwith4 g (a x) (b y) (c z) (d w)){ def. of zipwith, map }map f (zipwith4 g a b c d) map f (zipwith4 g x y z w)
{ hypo. of induction }zipwith4 g ′ (map f1 a) (map f2 b) (map f3 c) (map f4 d)
zipwith4 g ′ (map f1 x) (map f2 y) (map f3 z) (map f4 w){ def. of zipwith, map }zipwith4 g ′ (map f1 (a x)) (map f2 (b y)) (map f3 (c z)) (map f4 (d
−
◦

=

w))

−
◦

−
◦

−
◦

=

−
◦

−
◦

=

217

Bibliography
The inductive case for −◦ is proved similarly.

Rule VIII−◦

sum ◦ ( x) = (+(sum x)) ◦ sum
Proof:x) = sum y + sum x = ((+(sum x)) ◦ sum) y

−
◦

−
◦

(sum ◦ ( x)) y = sum (yRule IX−◦

map sum(map ( top ′ tr2 ) tr1 −◦ tr2 )
{ def. of map }◦ map sumtr2
map sum(map ( top ′ tr2 ) tr1 )−{ def. of map }◦ map sumtr2
map (sum ◦ ( top ′ tr2 )) tr1 −{ V, VIII }map (+ top ′ (map sum tr2 )) (map sum tr1 )−◦ map sumtr2
−
◦

=

−
◦

=
=

Rule X
reduce(⊕, ⊕)(zipwith(⊗) (zipwith(⊕) a b) (zipwith(⊕) c d))= reduce(⊕, ⊕)(zipwith(⊗) a c) ⊕ reduce(⊕, ⊕)(zipwith(⊗) a d)
⊕ reduce(⊕, ⊕)(zipwith(⊗) b c) ⊕ reduce(⊕, ⊕)(zipwith(⊗) b d)
⇐ (a ⊕ b) ⊗ (c ⊕ d) = (a ⊗ c) ⊕ (a ⊗ d) ⊕ (b ⊗ c) ⊕ (b ⊗ d)
Proof: It is proved by induction on the structure of abide trees.
reduce(⊕, ⊕)(zipwith(⊗) (zipwith(⊕) |a| |b|)) (zipwith(⊗) (zipwith(⊕) |c| |d|)){ def. of zipwith, reduce }(a ⊕ b) ⊗ (c ⊕ d)=
{ hypo. }(a ⊗ c) ⊕ (a ⊗ d) ⊕ (b ⊗ c) ⊕ (b ⊗ d)=
{ def. of zipwith, reduce }reduce(⊕, ⊕)(zipwith(⊗) |a| |c|) ⊕ reduce(⊕, ⊕)(zipwith(⊗) |a| |d|)⊕ reduce(⊕, ⊕)(zipwith(⊗) |b| |c|) ⊕ reduce(⊕, ⊕)(zipwith(⊗) |b| |d|)
=

218

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

reduce(⊕, ⊕)(zipwith(⊗) (zipwith(⊕) (a1 a2 ) (b1 b2 ))) (zipwith(⊗) (zipwith(⊕) (c1 c2 ) (d1 d2 )))={ def. of zipwith, reduce }reduce(⊕, ⊕)(zipwith(⊗) (zipwith(⊕) a1 b1 )) (zipwith(⊗) (zipwith(⊕) c1 d1 ))⊕ reduce(⊕, ⊕)(zipwith(⊗) (zipwith(⊕) a2 b2 )) (zipwith(⊗) (zipwith(⊕) c2 d2 ))={ hypo. of induction }reduce(⊕, ⊕)(zipwith(⊗) a1 c1 ) ⊕ reduce(⊕, ⊕)(zipwith(⊗) a1 d1 )⊕ reduce(⊕, ⊕)(zipwith(⊗) b1 c1 ) ⊕ reduce(⊕, ⊕)(zipwith(⊗) b1 d1 )⊕ reduce(⊕, ⊕)(zipwith(⊗) a2 c2 ) ⊕ reduce(⊕, ⊕)(zipwith(⊗) a2 d2 )⊕ reduce(⊕, ⊕)(zipwith(⊗) b2 c2 ) ⊕ reduce(⊕, ⊕)(zipwith(⊗) b2 d2 )={ def. of zipwith, reduce }reduce(⊕, ⊕)(zipwith(⊗) (a1 a2 ) (c1 c2 ))⊕ reduce(⊕, ⊕)(zipwith(⊗) (a1 a2 ) (d1 d2 ))⊕ reduce(⊕, ⊕)(zipwith(⊗) (b1 b2 ) (c1 c2 ))⊕ reduce(⊕, ⊕)(zipwith(⊗) (b1 b2 ) (d1 d2 ))The inductive case for −◦ is proved similarly.

Rule XI
max (map max (gemm( , zipwith(+)) b t))= max (zipwith(+) (reduce (zipwith(↑), ) b) (reduce ( , zipwith(↑)) t))⇐ width b = 1, height t = 1
Proof:It is proved by induction on the structure of abide trees.
===max (map max (gemm( , zipwith(+)) |b| |t|)){ def. of gemm }max (map max (| zipwith(+) b t|)){ def. of zipwith, map, max }max (zipwith(+) b t)
{ def. of reduce }max (zipwith(+) (reduce (zipwith(↑), ) |b|) (reduce ( , zipwith(↑)) |t|))−◦max (map max (gemm( , zipwith(+)) (b1 −◦ b2 ) (t1 t2 ))){ def. of gemm }max (map max (((gemm( , zipwith(+)) b1 t1 ) (gemm( , zipwith(+)) b1 t2 ))−◦((gemm( , zipwith(+)) b2 t1 ) (gemm( , zipwith(+)) b2 t2 ))))
{ def. of max }max (map max (gemm( , zipwith(+)) b1 t1 )) ↑max (map max (gemm( , zipwith(+)) b1 t2 ))↑ max (map max (gemm( , zipwith(+)) b2 t1 ))↑ max (map max (gemm( , zipwith(+)) b2 t2 )){ hypo. of induction }max (zipwith(+) (reduce (zipwith(↑), ) b1 ) (reduce ( , zipwith(↑)) t1 ))↑ max (zipwith(+) (reduce (zipwith(↑), ) b1 ) (reduce ( , zipwith(↑)) t2 ))↑ max (zipwith(+) (reduce (zipwith(↑), ) b2 ) (reduce ( , zipwith(↑)) t1 ))↑ max (zipwith(+) (reduce (zipwith(↑), ) b2 ) (reduce ( , zipwith(↑)) t2 )){ X with ⊕ =↑, ⊗ = + }◦ b2 )) (reduce ( , zipwith(↑)) (t1 t2 )))max (zipwith(+) (reduce (zipwith(↑), ) (b1 −

=

=

=

−
◦

−
◦

−
◦

=

219

Bibliography

Rule XII
max (zipwith4 fs s1 s2 r1 l2 )= max s1 ↑ max s2 ↑ max (zipwith(+) (map reduce(↑, ) r1 ) (map reduce( , ↑) l2 ))where fs s1 s2 r1 l2 = s1 ↑ max (gemm( , +) r1 l2 ) ↑ s2
⇐ width of elements of r1 = 1, height of elements of l2 = 1
Proof: First, we prove the following equation by the induction on the structure of abide trees.
max (zipwith4 fs s1 s2 r1 l2 ) = max s1 ↑ max s2 ↑ max (zipwith fs′ r1 l2 )where fs′ r1 l2 = max (gemm( , +) r1 l2 )Proof:=max (zipwith4 fs |s1 | |s2 | |r1 | |l2 |){ def. of fs , zipwith }s1 ↑ max (gemm( , +) r1 l2 ) ↑ s2
{ def. of fs′ , max , associativity of ↑ }max |s1 | ↑ max |s2 | ↑ max (zipwith fs′ |r1 | |l2 |)

=

−
◦

=

−
◦

−
◦

−
◦

−
◦

=

−
◦

−
◦

max (zipwith4 fs (s11 s21 ) (s12 s22 ) (r11 r12 ) (l21 l22 )){ def. of max , zipwith }max (zipwith4 fs s11 s12 r11 l21 ) ↑ max (zipwith4 fs s21 s22 r12 l22 ){ hypo. of induction }max s11 ↑ max s12 ↑ max (zipwith fs′ r11 l21 )↑ max s21 ↑ max s22 ↑ max (zipwith fs′ r12 l22 ){ def. of fs′ , max , associativity of ↑ }max (s11 s21 ) ↑ max (s12 s22 ) ↑ max (zipwith fs′ (r11 r12 ) (l21l22 ))−◦=The inductive case for −◦ is proved similarly.
Then, we prove the following equation by the induction on the structure of abide trees.
max (zipwith fs′ r1 l2 ) = max (zipwith(+) (map reduce(↑, ) r1 ) (map reduce( , ↑) l2 ))Proof:===max (zipwith fs′ |r1 | |l2 |){ def. of max , zipwith, fs′ }max | max (gemm( , +) r1 l2 )|{ below }max (| reduce(↑, ) r1 + reduce(↑, ) l2 |){ def. of zipwith, map }max (zipwith(+) (map reduce(↑, ) |r1 |) (map reduce( , ↑) |l2 |))

=

−
◦

=

−
◦

=

−
◦

−
◦

max (zipwith fs′ (r11 r12 ) (l21 l22 )){ def. of max , zipwith }max (zipwith fs′ r11 l21 ) ↑ max (zipwith fs′ r12 l22 ){ hypo. of induction }max (zipwith(+) (map reduce(↑, ) r11 ) (map reduce( , ↑) l21 ))↑ max (zipwith(+) (map reduce(↑, ) r12 ) (map reduce( , ↑) l22 )){ def. of max , zipwith, map }max (zipwith(+) (map reduce(↑, ) (r11 r12 )) (map reduce( , ↑) (l21 l22 )))220
To complete the proof of the base case, we prove the next equation by the induction on the
structure of abide trees.
max (gemm( , +) r1 l2 ) = reduce(↑, ) r1 + reduce(↑, ) l2
⇐ width r1 = 1, height l2 = 1
Proof:==max (gemm( , +) |r1 | |l2 |){ def. of gemm, max }r1 + l2{ def. of reduce }reduce(↑, ) |r1 | + reduce(↑, ) |l2 |−◦◦ r12 ) (l21 l22 ))max (gemm( , +) (r11 −{ def. of gemm, max }max (gemm( , +) r11 l21 ) ↑ max (gemm( , +) r11 l22 )↑ max (gemm( , +) r12 l21 ) ↑ max (gemm( , +) r12 l22 ){ hypo. of induction }(reduce(↑, ) r11 + reduce(↑, ) l21 ) ↑ (reduce(↑, ) r11 + reduce(↑, ) l22 )↑ (reduce(↑, ) r12 + reduce(↑, ) l21 ) ↑ (reduce(↑, ) r12 + reduce(↑, ) l22 ){ associativity and distributivity }(reduce(↑, ) r11 ↑ reduce(↑, ) r12 ) + (reduce(↑, ) l21 ↑ reduce(↑, ) l22 ){ def. of reduce }reduce(↑, ) (r11 −◦ r12 ) + reduce(↑, ) (l21 l22 )

=

=

=

−
◦

=

Rule XIII
reduce(⊕, ⊗) (map f x)
= f (reduce(⊕, ⊗) x) ⇐ f a ⊗ f b = f (a ⊗ b), f a ⊕ f b = f (a ⊕ b)
Proof: It is proved by induction on the structure of abide trees.
=
=

=
=
=

reduce(⊕, ⊗) (map f (x−◦ y)){ def. of reduce, map }reduce(⊕, ⊗) (map f x)) ⊕ reduce(⊕, ⊗) (map f y)
{ hypo. of induction }f (reduce(⊕, ⊗) x) ⊕ f (reduce(⊕, ⊗) y){ hypo. }f ((reduce(⊕, ⊗) x) ⊕ (reduce(⊕, ⊗) y)){ def. of reduce }f (reduce(⊕, ⊗) (x y))−◦=

reduce(⊕, ⊗) (map f |x|){ def. of reduce, map }f x{ def. of reduce }f (reduce(⊕, ⊗) |x|)

The inductive case for −◦ is proved similarly.
For instance, ⊕ = (don’t care)，⊗ = zipwith(↑) and f = zipwith(+) c1 satisfy the conditionf a ⊗ f b = f (a ⊗ b) .

221

Bibliography

Rule XIV
reduce(⊕, ⊗) (zipwith4 f x y z w)
= f ′ (reduce(⊕1 , ⊗1 ) x) (reduce(⊕2 , ⊗2 ) y) (reduce(⊕3 , ⊗3 ) z) (reduce(⊕4 , ⊗4 ) w)
⇐ f a b c d = f ′ a b c d,f ′ a b c d ⊕ f ′ x y z w = f ′ (a ⊕1 x) (b ⊕2 y) (c ⊕3 z) (d ⊕4 w)
f ′ a b c d ⊗ f ′ x y z w = f ′ (a ⊗1 x) (b ⊗2 y) (c ⊗3 z) (d ⊗4 w)
Proof: It is proved by induction on the structure of abide trees.
reduce(⊕, ⊗) (zipwith4 f |x| |y| |z| |w|)
{ def. of reduce, zipwith }f xyzw={ hypo. }f′ x y z w={ def. of reduce }f ′ (reduce(⊕1 , ⊗1 ) |x|) (reduce(⊕2 , ⊗2 ) |y|) (reduce(⊕3 , ⊗3 ) |z|) (reduce(⊕4 , ⊗4 ) |w|)=

reduce(⊕, ⊗) (zipwith4 f (x1 −◦ x2 ) (y1 −◦ y2 ) (z1 −◦ z2 ) (w1 −◦ w2 )){ def. of reduce, zipwith }reduce(⊕, ⊗) (zipwith4 f x1 y1 z1 w1 ) ⊕ reduce(⊕, ⊗) (zipwith4 f x2 y2 z2 w2 )={ hypo. of induction }f ′ (reduce(⊕1 , ⊗1 ) x1 ) (reduce(⊕2 , ⊗2 ) y1 ) (reduce(⊕3 , ⊗3 ) z1 ) (reduce(⊕4 , ⊗4 ) w1 )⊕f ′ (reduce(⊕1 , ⊗1 ) x2 ) (reduce(⊕2 , ⊗2 ) y2 ) (reduce(⊕3 , ⊗3 ) z2 ) (reduce(⊕4 , ⊗4 ) w2 )={ hypo. }f ′ (reduce(⊕1 , ⊗1 ) x1 ⊕1 reduce(⊕1 , ⊗1 ) x2 ) (reduce(⊕2 , ⊗2 ) y1 ⊕2 reduce(⊕2 , ⊗2 ) y2 )(reduce(⊕3 , ⊗3 ) z1 ⊕3 reduce(⊕3 , ⊗3 ) z3 ) (reduce(⊕4 , ⊗4 ) w1 ⊕4 reduce(⊕4 , ⊗4 ) w2 )={ def. of reduce }f ′ (reduce(⊕1 , ⊗1 ) (x1 −◦ x2 )) (reduce(⊕2 , ⊗2 ) (y1 −◦ y2 ))(reduce(⊕3 , ⊗3 ) (z1 −◦ z2 )) (reduce(⊕4 , ⊗4 ) (w1 −◦ w2 ))=

Rule XV−◦−◦

map (reduce(⊕, )) (gemm( , zipwith(⊗)) x y)
=gemm(⊕, ⊗) (tr (reduce ( , )) x) (reduce ( , ) y)
⇐ width of x and its elements = 1, width of y’s elements = 1, height y = 1
Proof: It is proved by induction on the structure of abide trees.
map (reduce(⊕, )) (gemm( , zipwith(⊗)) |x| |y|){ def. of map, gemm }| reduce(⊕, ) x y|=
{ below }gemm (⊕, ⊗) (tr x) y=
{ def. of reduce }gemm (⊕, ⊗) (tr (reduce ( , ) |x|)) (reduce ( , ) |y|)
−
◦

−
◦

=

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

The inductive case for is proved similarly.
◦ (NIL b), ⊗1 = zipwith(↑), ⊗2 = zipwith(↑),
For instance, f ′ a b c d = (a gemm(↑, +) c d) −⊗3 =and ⊗4 = −◦ satisfy the condition for f a b c d = (a gemm( , +) c d) −◦ (NIL b),⊗ = zipwith(↑) and ⊕ = .

222
−
◦

◦ x2 ) (y1 y2 ))map (reduce(⊕, )) (gemm( , zipwith(⊗)) (x1 −{ def. of map, gemm }(map (reduce(⊕, ))(gemm( , zipwith(⊗)) x1 y1 )map (reduce(⊕, )) (gemm( , zipwith(⊗)) x1 y2 ))−◦(map (reduce(⊕, )) (gemm( , zipwith(⊗)) x2 y1 )map (reduce(⊕, )) (gemm( , zipwith(⊗)) x2 y2 ))={ hypo. of induction }(gemm (⊕, ⊗) (tr (reduce ( , ) x1 )) (reduce ( , ) y1 )gemm (⊕, ⊗) (tr (reduce ( , ) x1 )) (reduce ( , ) y2 ))−◦(gemm (⊕, ⊗) (tr (reduce ( , ) x2 )) (reduce ( , ) y1 )gemm (⊕, ⊗) (tr (reduce ( , ) x2 )) (reduce ( , ) y2 ))={ def. of gemm }gemm (⊕, ⊗) (tr (reduce ( , ) x1 ) tr (reduce ( , ) x2 )) (reduce ( , ) y1=
{ def. of tr , reduce }gemm (⊕, ⊗) (tr (reduce ( , ) (x1 −◦ x2 ))) (reduce ( , ) (y1 y2 ))
−
◦

−
◦

−
◦

reduce ( , ) y2 ))

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

To complete the proof, we prove the following equation by the induction on the structure ofabide trees.
| reduce(⊕, ) (zipwith(⊗) x y)| = gemm(⊕, ⊗) (tr x) y
⇐ width x = 1, width y = 1
Proof:

=
=

=
=
=
=

| reduce(⊕, ) (zipwith(⊗) |x| |y|)|{ def. of zipwith, reduce }|x ⊗ y|{ def. of gemm, tr }gemm(⊕, ⊗) (tr |x|) |y|| reduce(⊕, ) (zipwith(⊗) (x1 −◦ x2 ) (y1 −◦ y2 ))|{ def. of zipwith, reduce }| reduce(⊕, ) (zipwith(⊗) x1 y1 ) ⊕ reduce(⊕, ) (zipwith(⊗) x2 y2 )|{ def. of zipwith }zipwith(⊕) | reduce(⊕, ) (zipwith(⊗) x1 y1 )| | reduce(⊕, ) (zipwith(⊗) x2 y2 )|{ hypo. of induction }zipwith(⊕) (gemm(⊕, ⊗) (tr x1 ) y1 ) (gemm(⊕, ⊗) (tr x2 ) y2 ){ def. of gemm, tr }gemm(⊕, ⊗) (tr (x1 −◦ x2 )) (y1 −◦ y2 )

Rule XVI
map (reduce( , ⊕)) (gemm( , zipwith(⊗)) x y)
= gemm(⊕, ⊗) (reduce (−◦, ) x) (tr (reduce ( ,−◦) y))⇐ width x = 1, height of x’s elements = 1, height of y and its elements = 1
Proof: Similar to Rule XV.

223

Bibliography

Rule XVII−◦

map(reduce(↑, )) (zipwith3 fr r1 r2 ro2 )= zipwith3 fr′ (reduce (↑) r1 ) ro2 (reduce (↑, )r2 )where fr r1 r2 ro2 = map (+ro2 ) r1 r2
fr′ r1 ro2 r2 = (r1 + ro2 ) ↑ r2
Proof: Rule VII and following calculation.
reduce(↑, ) (fr r1 r2 ro2 ){ def. of fr }reduce(↑, ) ((map (+ro2 )r1 ) r2 ){ def. of reduce }reduce(↑, ) (map (+ro2 )r1 ) ↑ r2
{ + distributes over ↑ }((reduce(↑, ) r1 ) + ro2 ) ↑ r2
−
◦

=
=
=

Rule XVIII
−
◦

−
◦

−
◦

reduce( , ) (map (zipwith(+) (right ′ x)) y)= mapc (zipwith(+) (right (reduce( , ) x))) (reduce( , ) y)
⇐ height x = 1, width of x’s elements = 1
Proof: First, we prove the next equation by the induction on the structure of abide trees.
−
◦

−
◦

reduce( , ) (map f x) = mapc f (reduce( , ) x)
⇐ height x = 1, width of x’s elements = 1
Proof:
reduce( , ) (map f |x|){ def. of reduce, map }f x{ def. of mapc , height x = 1 }mapc f x{ def. of reduce }mapc f (reduce( , ) |x|)
−
◦

=
=

−
◦

=

−
◦

−
◦

reduce( , ) (map f (x1 x2 )){ def. of reduce, map }map f x1 map f x2
{ hypo. of induction }mapc f (reduce( , ) x1 ) mapc f (reduce( , ) x2 ){ def. of mapc }mapc f (reduce( , ) (x1 x2 ))
−
◦

−
◦
−
◦

=

−
◦

=

−
◦

−
◦

=

To complete the proof, we prove the next equation by the induction on the structure of abide
trees.
−◦

right ′ x = right (reduce( , ) x)
⇐ height x = 1, width of x’s elements = 1

224
Proof:==−◦=

right ′ |x|{ def. of right ′ }x{ def. of right, width x = 1 }right x{ def. of reduce }right (reduce( , ) |x|)−◦

right ′ (x1 x2 ){ def. of right ′ }right ′ x2
{ hypo. of induction }right (reduce( , ) x2 ){ def. of right }right (reduce( , ) x1 reduce( , ) x2 ){ def. of reduce }right (reduce( , ) (x1 x2 ))

=

−
◦

−
◦

−
◦

=

−
◦

=

−
◦

−
◦

=

Rule XIX
−
◦

−
◦

−
◦

reduce( , ) (map (zipwith(+) (top ′ x)) y)= mapc (zipwith(+) (top (reduce( , ) x))) (reduce( , ) y)
⇐ width x = 1, width of x’s elements = 1
Proof: Similar to Rule XVIII.

Rule XX
−
◦

−
◦

−
◦

Proof: Rule XIV with f ′ a b = mapr (zipwith(+) (top b)) a−◦ b and ⊗ = , ⊗1 = , ⊗2 =

.

−
◦

−
◦

−
◦

−
◦

reduce( , ) (zipwith f x y)
= mapr (zipwith(+) (top (reduce( , ) y))) (reduce( , ) x)−◦ (reduce( , ) y)
⇐ height x = 1, height y = 1, width of x and y’s elements = 1
f x y = map (+(top ′ y)) x−◦y

Rule XXI
−
◦

−
◦

−
◦

Proof: Rule XIV with f ′ a b = mapr (zipwith(+) (top b)) a−◦ b and ⊕ = , ⊕1 = , ⊕2 =

.

−
◦

−
◦

−
◦

−
◦

reduce( , ) (zipwith f x y)
= mapr (zipwith(+) (top (reduce( , ) y))) (reduce( , ) x)−◦ (reduce( , ) y)
⇐ width x = 1, width y = 1, width of x and y’s elements = 1
f x y = map (+(top ′ y)) x−◦yAppendix B
Complete Proof of Theorem 5.1
In the following, we use an abbreviation of the index accessing for readability: hxii = at x i.
We assume that the length of an input list is n. The goal of this proof is to show the equation
evalP prog = eval (compile prog) for any prog.

Base Case
What we have to prove is the following equation for a list x.
evalP x =eval [[ [], Leafc id (Var x 0), [] ]](B.1)

The ith element of the left-hand sides is as follows.
hevalP xii={ definition of evalP }hevalP xii

The ith element of the right-hand sides is as follows.
heval [[ [], Leafc id (Var x 0), [] ]]ii

=
=
=
=

{ definition of eval }hmap evalT (Leafc id (Var x 0)) [0..n − 1]ii
{ ith element }evalT (Leafc id (Var x 0)) i{ definition of evalT and the identity function }evalV (Var x 0) i
{ definition of evalV and ith element }hxii

Thus, the following equation holds.
hevalP xii=

heval [[ [], Leafc id (Var x 0), [] ]]ii

Since this equation holds for i ∈ [0..n − 1], the equation (B.1) holds.

Inductive Case for map
What we have to prove is the following equation for a function f and a program prog. Here,compile prog = [[ ls, zms, rs ]].
evalP (map f prog)=

eval [[ map (comp f ) ls, comp f zms, map (comp f ) rs ]]To prove this equation, we first show a lemma.

(B.2)

226
Lemma B.3. Let f be a function, t be a computational tree, and i be an index. Then, thefollowing equation holds.
evalT (comp f t) i=f (evalT t i)

Proof. This is shown by induction on Tree.
The Node case:evalT (comp f (Node g l r)) i

=
=
=
=

{ definition of comp }evalT (Node (f ◦ g) l r) i
{ definition of evalT }(f ◦ g) (evalT l i, evalT r i)
{ function composition }f (g (evalT l i, evalT r i)){ definition of evalT }f (evalT (Node g l r) i)

The Leafv case:evalT (comp f (Leafv g v)) i

=
=
=
=

{ definition of comp }evalT (Leafv (f ◦ g) v) i
{ definition of evalT }(f ◦ g) (evalV v i)
{ function composition }f (g (evalV v i)){ definition of evalT }f (evalT (Leafv g v) i)

The Leafc case:evalT (comp f (Leafc c)) i={ definition of comp }evalT (Leafc (f c)) i={ definition of evalT }f c={ definition of evalT }f (evalT (Leafc c) i)

Now, we show the equation (B.2). In the following, l = length ls, r = length rs, and idces =[l..(n − r − 1)].
=
=
=
=
=
=

evalP (map f prog){ definition of evalP }map f (evalP prog){ induction hypothesis }map f (eval [[ ls, zms, rs ]]){ definition of eval }map f (map evalT0 ls ++ map (evalT zms) idces ++ map evalT0 rs){ definition of map and its distributivity: map h ◦ map g = map (h ◦ g) }map (f ◦ evalT0 ) ls ++ map (f ◦ evalT zms) idces ++ map (f ◦ evalT0 ) rs
{ Lemma B.3 and definition of evalT0 }map (evalT0 ◦comp f ) ls ++ map (evalT (comp f zms)) idces ++ map (evalT0 ◦comp f ) rs
{ distributivity of map and definition of eval }eval [[ map (comp f ) ls, comp f zms, map (comp f ) rs ]]Thus, the equation (B.2) holds.

227

Bibliography

Inductive Cases for shift and shift≪≫

What we have to prove is the following equations for an element e and a program prog. Here,compile prog = [[ ls, zms, rs ]].
evalP (shift≪ e prog) = eval [[ tail ls, slide (−1) zms, rs ++ [Leafc e] ]]evalP (shift≫ e prog) = eval [[ [Leafc e] ++ ls, slide 1 zms, init rs ]]

(B.4)
(B.5)

To prove this equation, we first show a lemma.

Lemma B.6. Let d be an integer, i be an index, and t be a computational tree. Then, thefollowing equation holds.
evalT (slide d t) i=evalT t (i − d)(B.7)

Proof. This is shown by induction on Tree and Var .
The Node case:evalT (slide d (Node f l r)) i

=
=
=
=

{ definition of slide }evalT (Node f (slide d l) (slide d r))) i{ definition of evalT }f (evalT (slide d l) i, evalT (slide d r) i){ induction hypothesis }f (evalT l (i − d), evalT r (i − d)){ definition of evalT }evalT (Node f l r) (i − d)

The Leafv with Var case:evalT (slide d (Leafv f (Var x s))) i

=
=
=
=

{ definition of slide }evalT (Leafv f (Var x (s + d))) i{ definition of evalT and evalV }f (at x (i − (s + d))){ arithmetic }f (at x ((i − d) − s)){ definition of evalT and evalV }evalT (Leafv f (Var x s)) (i − d)Since other cases of Leafv and the case of Leafc ignore the index i, the equation (B.7) holds inthese cases.
Thus, the equation (B.7) holds.

Now, we show the equation (B.4). In the following, l = length ls and r = length rs. First, we228
assume ls is not empty, i.e. l > 0.
=
=
=
=
=

=

=

evalP (shift≪ e prog){ definition of evalP }shift≪ e (evalP prog){ induction hypothesis }shift≪ e (eval [[ ls, zms, rs ]]){ definition of eval }shift≪ e (map evalT0 ls ++ map (evalT zms) [l..(n − r − 1)] ++ map evalT0 rs){ the relation shift≪ e x = tail x ++ [e], and ls being not empty }tail (map evalT0 ls) ++ map (evalT zms) [l..(n − r − 1)] ++ map evalT0 rs ++ [e]{ definition of map and evalT0 }map evalT0 (tail ls) ++ map (evalT zms) (map (+1) [(l − 1)..(n − (r + 1) − 1)])++ map evalT0 (rs ++ [Leafc e]){ distributivity of map, and Lemma B.6 }map evalT0 (tail ls) ++ map (evalT (slide (−1) zms)) [l′ ..(n − r′ − 1)]++ map evalT0 (rs ++ [Leafc e])
where l′ = length tail ls, r′ = length (rs ++ [Leafc e]){ definition of eval }eval [[ tail ls, slide (−1) zms, rs ++ [Leafc e] ]]Next, we assume ls = [ ].
=
=
=
=
=

=

evalP (shift≪ e prog){ the same as the previous calculation }shift≪ e (map evalT0 ls ++ map (evalT zms) [l..(n − r − 1)] ++ map evalT0 rs){ the relation shift≪ e x = tail x ++ [e], and ls being empty }tail (map (evalT zms) [0..(n − r − 1)]) ++ map evalT0 rs ++ [e]{ definition of tail }map (evalT zms) ([1..(n − r − 1)]) ++ map evalT0 rs ++ [e]{ definition of map and evalT0 }map (evalT zms) (map (+1) [0..(n − (r + 1) − 1)]) ++ map evalT0 (rs ++ [Leafc e]){ distributivity of map, Lemma B.6, and tail ls = [ ] }map evalT0 (tail ls) ++ map (evalT (slide (−1) zms)) [l′ ..(n − r′ − 1)]++ map evalT0 (rs ++ [Leafc e])
where l′ = tail ls, r′ = length (rs ++ [Leafc e]){ definition of eval }eval [[ tail ls, slide (−1) zms, rs ++ [Leafc e] ]]Thus, the equation (B.4) holds.
The equation (B.5) is shown similarly.

Inductive Case for zip
What we have to prove is the following equation for two programs prog 1 and prog 2 .
compile prog 1 = [[ ls 1 , zms 1 , rs 1 ]] and compile prog 2 = [[ ls 2 , zms 2 , rs 2 ]].
evalP (zip prog 1 prog 2 ) = eval [[ ls, zms, reverse rs ]]where zms = Node id zms 1 zms 2
ls = trim FromL ls 1 ls 2 zms 1 zms 2
rs = trim FromR (reverse rs 1 ) (reverse rs 2 ) zms 1 zms 2
To prove this equation, we first show a lemma.

Here,
(B.8)

229

Bibliography

Lemma B.9. Let i be an index, and t be a computational tree. Then, the following equation
holds.
evalT0 (inst FromL i)=

evalT t i

(B.10)

Proof. This is shown by induction on Tree and Var .
The Node case:evalT0 (inst FromL (Node f l r) i)

=
=
=
=

{ definition of inst }evalT0 (Node f (inst FromL l i) (inst FromL r i)){ definition of evalT0 }f (evalT0 (inst FromL l i), evalT0 (inst FromL r i)){ induction hypothesis }f (evalT l i, evalT r i)
{ definition of evalT }evalT (Node f l r) i

The Leafv with Var case:evalT0 (inst FromL (Leafv f (Var x s)) i)==={ definition of inst }evalT0 (Leafv f (Fix x (−s + i) FromL)){ definition of evalT0 }f (at x (−s + i)){ definition of evalT }evalT (Var x s) i

Since other cases of Leafv and the case of Leafc merely return the given tree and this tree ignores
the index, the equation (B.10) holds in these cases.
Thus, the equation (B.10) holds.

We also use the following lemma.

Lemma B.11. Let i be an index, and t be a computational tree. Then, the following equation
holds.
evalT0 (inst FromR i)=

evalT t (n − 1 − i)

Proof. Similar to the proof of Lemma B.9.

In the following, l1 = length ls 1 , r1 = length rs 1 , l2 = length ls 2 and r2 = length rs 2 . First,

230
we assume l1 ≥ l2 and r1 ≥ r2 .
=
=
=

=

=

evalP (zip prog 1 prog 2 ){ definition of evalP }zip (evalP prog 1 , evalP prog 2 ){ induction hypothesis }zip (eval [[ ls 1 , zms 1 , rs 1 ]]) (eval [[ ls 2 , zms 2 , rs 2 ]]){ definition of eval and zip, letting ls 1 = ls 11 ++ ls 12 and rs 1 = rs 11 ++ rs 12 }zip (map evalT0 ls 11 ) (map evalT0 ls 2 )++ zip (map evalT0 ls 11 ) (map (evalT zms 2 ) [l2 ..(l1 − 1)])++ zip (map (evalT zms 1 ) [l1 ..(n − r1 − 1)]) (map (evalT zms 2 ) [l1 ..(n − r1 − 1)])++ zip (map evalT0 rs 11 ) (map (evalT zms 2 ) [(n − r1 )..(n − r2 − 1)])++ zip (map evalT0 rs 12 ) (map evalT0 rs 2 ){ Lemma B.9 and Lemma B.11 }zip (map evalT0 ls 11 ) (map evalT0 ls 2 )++ zip (map evalT0 ls 11 ) (map evalT0 (map (inst FromL zms 2 ) [l2 ..(l1 − 1)])++ zip (map (evalT zms 1 ) [l1 ..(n − r1 − 1)]) (map (evalT zms 2 ) [l1 ..(n − r1 − 1)])++ zip (map evalT0 rs 11 ) (map evalT0 (map (inst FromR zms 2 ) [(r1 − 1)..r2 ])++ zip (map evalT0 rs 12 ) (map evalT0 rs 2 ){ introducing (Node id ), and definition of eval and evalT0 }map evalT0 (zipwith (Node id ) ls 11 ls 2 )++ map evalT0 (zipwith (Node id ) ls 11 (map (inst FromL zms 2 ) [l2 ..(l1 − 1)]))++ map (evalT (Node id zms 1 zms 2 )) [l1 ..(n − r1 − 1)]++ map evalT0 (zipwith (Node id ) rs 11 (map (inst FromR zms 2 ) [(r1 − 1)..r2 ]))++ map evalT0 (zipwith (Node id ) rs 12 rs 2 )={ combining edge elements }map evalT0 (zipwith (Node id ) ls 1 (ls 2 ++ map (inst FromL zms 2 ) [l2 ..(l1 − 1)]))++ map (evalT (Node id zms 1 zms 2 )) [l1 ..(n − r1 − 1)]++ map evalT0 (zipwith (Node id ) rs 1 (map (inst FromR zms 2 ) [(r1 − 1)..r2 ] ++ rs 2 ))={ definition of eval and trim }eval [[ ls, zms, reverse rs ]]where zms = Node id zms 1 zms 2
ls = trim FromL ls 1 ls 2 zms 1 zms 2
rs = trim FromR (reverse rs 1 ) (reverse rs 2 ) zms 1 zms 2
The other cases (l1 ≥ l2 ∧ r1 < r2 , l1 < l2 ∧ r1 ≥ r2 and l1 < l2 ∧ r1 < r2 ) are similarly shown.
Thus, the equation (B.8) holds.

Appendix C
Auxiliary Rules for the Proof of
Theorem 5.44
Rule I
shift≪ e = flatten ◦ shift≪ [e] ◦ map [·]Proof: It is shown by the following calculation. Some where-clauses are omitted for readability.

=

=
=
=
=
=

shift≪ e{ definition of shift≪ }map π1 ◦ scanr′ (⊕) ( , e, ) ◦ map f
where f a = (a, a, True)( , c, True) ⊕ ( , p, ) = (p, c, False)(p, c, False) ⊕ ( , , ) = (p, c, False){ flatten ◦ map [·] = id }flatten ◦ map [·] ◦ map π1 ◦ scanr′ (⊕) ( , e, ) ◦ map f{ [·] ◦ π1 = π1 ◦ ([·] × [·] × id ) }flatten ◦ map π1 ◦ map ([·] × [·] × id ) ◦ scanr′ (⊕) ( , e, ) ◦ map f{ ([·]×[·]×id ) x ⊕ ([·]×[·]×id ) y = ([·]×[·]×id ) (x ⊕ y), and fusion of map and scanr′ }flatten ◦ map π1 ◦ scanr′ (⊕) ( , [e], ) ◦ map ([·] × [·] × id ) ◦ map f{ ([·] × [·] × id ) ◦ f = f ◦ [·] }flatten ◦ map π1 ◦ scanr′ (⊕) ( , [e], ) ◦ map f ◦ map [·])
{ definition of shift≪ }flatten ◦ shift≪ [e] ◦ map [·]Rule II
flatten ◦ shift≪ y ◦ map [·] = (++y) ◦ flatten ◦ shift≪ [ ] ◦ map [·]232
Proof: It is shown by the following calculation. Some where-clauses are omitted for readability.
(++y) ◦ flatten ◦ shift≪ [ ] ◦ map [·]{ definition of shift≪ }(++y) ◦ flattenmap π1 ◦ ◦map (⊕( , [ ], )) ◦ scanr (⊕) ◦ map f ◦ map [·]where f a = (a, a, True)( , c, True) ⊕ ( , p, ) = (p, c, False)(p, c, False) ⊕ ( , , ) = (p, c, False)= { (++y) ◦ flatten = flatten ◦ (++[y]) }flatten ◦ (++[y]) ◦ map π1 ◦ ◦map (⊕( , [ ], )) ◦ scanr (⊕) ◦ map f ◦ map [·]= { only the last element in the result of scanr (⊕) has True in the third component }flattenmap π1 ◦ map (⊕( , y, )) ◦ scanr (⊕) ◦ map f ◦ map [·]= { definition of shift≪ }flatten ◦ shift≪ y ◦ map [·]=Rule III
shift≪ e (x ++ y) = shift≪ (head y) x ++ shift≪ e y
Proof: It is shown by the following calculation. Some where-clauses are omitted for readability.

=

=

=

=

shift≪ e (x ++ y){ definition of shift≪ }(map π1 ◦ map (⊕( , e, )) ◦ scanr (⊕) ◦ map f ) (x ++ y)where f a = (a, a, True)( , c, True) ⊕ ( , p, ) = (p, c, False)(p, c, False) ⊕ ( , , ) = (p, c, False){ definition of map and scanr }(map π1 ◦ map (⊕( , e, )) ◦ map (⊕(reduce (≪) (scanr (⊕) (map f y))) ◦ scanr (⊕) ◦ map f ) x++(map π1 ◦ map (⊕( , e, )) ◦ scanr (⊕) ◦ map f ) y{ definition of ⊕ }(map π1 ◦ map (⊕( , reduce (≪) y, )) ◦ scanr (⊕) ◦ map f ) x++(map π1 ◦ map (⊕( , e, )) ◦ scanr (⊕) ◦ map f ) y{ definition of shift≪ and head }shift≪ (head y) x ++ shift≪ e yRule IV
tail (x ++ y) = tail x ++ yProof: It is shown by the following induction on y. Some where-clauses are omitted forreadability.

233

Bibliography
The base case is shown as follows.
=

=
=

=
=
=

tail (x ++ [a]){ definition of tail and shift≪ }(flatten ◦ map π1 ◦ map (⊕( , [ ], )) ◦ scanr (⊕) ◦ map f ◦ map [·]) (x ++ [a])where f a = (a, a, True)( , c, True) ⊕ ( , p, ) = (p, c, False)(p, c, False) ⊕ ( , , ) = (p, c, False){ definition of map }(flatten ◦ map π1 ◦ map (⊕( , [ ], ))) (scanr (⊕) (map f (map [·] x)) ++ [([a], [a], True)]){ definition of scan }(flatten ◦ map π1 ◦ map (⊕( , [ ], ))) ((map (⊕([a], [a], True)) (scanr (⊕) (map f (map [·] x))))++[([a], [a], True)]){ ([a], [a], True) ⊕ ( , [ ], ) = ([ ], [a], False) }(flatten ◦ map π1 ) ((map (⊕( , [a], )) (scanr (⊕) (map f (map [·] x)))) ++ [([ ], [a], False)]){ definition of map and π1 }(flatten ◦ map π1 ) ((map (⊕( , [a], )) (scanr (⊕) (map f (map [·] x))))){ definition of shift≪, and rule II }tail x ++ [a]The induction case is shown as follows.
==

tail (x ++ (y ++ z)){ associativity of ++, and induction hypothesis }tail (x ++ y) ++ z
{ induction hypothesis, and associativity of ++ }tail x ++ (y ++ z)

Rule V[head x] ++ tail x = x
Proof: It is shown by the following induction on x.
The base case is shown as follows.
==[head [a]] ++ tail [a]{ definition of head and tail }[a] ++ [ ]{ the empty list }[a]

The induction case is shown as follows.
==[head (x ++ y)] ++ tail (x ++ y){ definition of head , and rule IV }[head x] ++ tail x ++ y
{ induction hypothesis }x ++ yRule VI
shift≪ e x = tail x ++ [e]Proof: It is shown by the following induction on x.

234

=
=
=
=

shift≪ e x{ rule I }(flatten ◦ shift≪ [e] ◦ map [·]) x{ rule II }((++[e]) ◦ flatten ◦ shift≪ [ ] ◦ map [·]) x{ definition of tail }((++[e]) ◦ tail) x{ function composition }tail x ++ [e]Appendix D
Dilation of Rects
We will show the following dilation of rects ′ .
flatten ◦ unwind−◦ ◦ map unwind|a||a| ⊕ reduce ( , ⊕) (map | · | x)
unwind ⊕ (|a| x) ⊕ unwind ⊕ y

−
◦

=
=
=
=

−
◦

y))

−
◦

◦
−
◦ −

rects ′ = pack ◦ rects
where packunwind ⊕ |a|unwind ⊕ (|a| x)
unwind ⊕ ((|a| x)−◦ (NILWe will first show the following dilation of TLs by induction. The result will be used in the
proof of the dilation of rects ′ .
◦) ◦ toplefts
TLs = reduce ( ,−The base case is shown below.
===

TLs |a|{ definition of TLs }||a||{ definition of reduce }reduce ( ,−◦) |||a|||{ definition of toplefts }◦) (toplefts |a|)reduce ( ,−The induction case of −◦ is as follows.
=
=

=

=

TLs (x−◦ y){ definition of TLs and scan }TLs◦ mapr (zipwith (−◦) (bottom (TLs x))) (TLs y) x−
induction hypothesis, and bottom ◦ TLs = right ′ ◦ toplefts
reduce ( ,−◦) (toplefts x)−◦ mapr (zipwith (−◦) (right ′ (reduce ( ,−◦) ( toplefts x)))) (reduce ( ,−◦) (toplefts y)){ definition of reduce, and a row-wise map is changed to a simple map along }reduce ( ,−◦) (toplefts x)−◦) ( toplefts x)))) (reduce ( ,−◦) (toplefts y))◦ map (zipwith (−◦) (right ′ (reduce ( ,−{ definition of toplefts }reduce ( ,−◦) (toplefts (x−
◦ y))
−
◦

The induction case of is similarly shown.
Now, we proceed to the dilation of rects ′ . We will show it by induction.

236
The singleton case is shown below.
===

rects ′ |a|{ definition of rects ′ }|||a|||{ definition of pack and unwind }pack |||a|||{ definition of rects }pack (rects |a|)

−
◦

−
◦

Next, we will show the induction case of x y. Here, we assume that width x = 1 for simplicityof the proof. Note that we can impose this assumption safely because the properties of constructors−◦ and allow restructuring of the input to satisfy the assumption.
The left hand side is calculated as follows.
flatten (map TLs (BRs (x y))){ definition of BRs }flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs y))) (BRs x) BRs y)){ definition of map and flatten }flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs y))) (BRs x)))flatten (map TLs (BRs y))
−
◦

−
◦

−
◦

=

−
◦

−
◦

=

The right hand side is as follows. We will omit redundant where-clauses for readability.
pack (rects (x y)){ definition of pack and rects }(flatten ◦ unwind−◦ ◦ map unwind ) (zipwith4 fs (rects x) (rects y) (rights x) (lefts y))◦ (NIL s2 )where fs s1 s2 r1 l2 = (s1 gemm ( , ) r1 l2 )−{ rects x and rights x are singletons }′flatten (unwind−◦ (zipwith4 fs (rects x) (rects y) (rights x) (lefts y))′where fs s1 s2 r1 l2 = (s1 map (the r1 ) l2 ) unwind s2 ){ split of zipwith4 }′′flatten (unwind−◦ (zipwith ( ) (zipwith3 fs (rects x) (rights x) (lefts y)))(map unwind (rects y)))where fs′′ s1 r1 l2 = s1 map (the r1 ) l2 ){ swap of unwind and zipwith }′′flatten (zipwith ( ) (unwind−◦ (zipwith3 fs (rects x) (rights x) (lefts y)))(unwind−(rects y)))◦ (map unwind{ flatten (zipwith ( ) X Y ) = flatten X flatten Y when width X = 1 }′′flatten (unwind−◦ (zipwith3 fs (rects x) (rights x) (lefts y)))flatten (unwind−(rects y)))◦ (map unwind
−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

Since we can use the induction hypothesis(rects y))),−◦flatten (map TLs (BRs y)) = flatten (unwind−◦ (map unwind
the rest of the proof is to show the following equation.

We will show this equation by induction on x.

−
◦

−
◦

−
◦

flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs y))) (BRs x)))′′= flatten (unwind−◦ (zipwith3 fs (rects x) (rights x) (lefts y)))′′where fs s1 r1 l2 = s1 map (the r1 ) l2 )

237

Bibliography
The base case (x = |a|) is shown as follows. Note that height y = 1 for the consistency.
−◦

flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs y))) (BRs |a|))){ definition of BRs }flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs y))) ||a||)){ left (BRs y) = y because height y = 1 }flatten (map TLs (mapc (λt. zipwith ( ) t y) ||a||)){ definition of mapc }flatten (map TLs ||a| y|){ definition of map and TLs }flatten (|||a|| mapr (zipwith ( ) |a|) (TLs y)|){ change of mapr to map from height (TLs y) = 1, and definition of flatten }||a|| map (|a| ) (TLs y){ the dilation of TLs }||a|| map (|a| ) (reduce ( ,−◦) (toplefts y)){ toplefts y = lefts y when height y = 1 from definition of toplefts and lefts }||a|| map (|a| ) (reduce ( ,−◦) (toplefts y))◦) |a| = a = the a, and lefts y is a singleton }{ reduce ( ,−◦) (lefts y))||a|| map (|a| ) (reduce ( ,−{ definition of flatten and unwind }◦) (lefts y))|)flatten (unwind−◦ |||a|| map (|a| ) (reduce ( ,−{ definition of zipwith3 and fs′′ }′′flatten (unwind−◦ (zipwith3 fs |||a||| |||a||| (lefts y)))′′where fs s1 r1 l2 = s1 map (the r1 ) l2 ){ definition of rects and rights }′′flatten (unwind−◦ (zipwith3 fs (rects |a|) (rights |a|) (lefts y)))
−
◦

=

−
◦

=

=

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

=

−
◦

=

−
◦

−
◦

=

−
◦

−
◦

=

−
◦

−
◦

=

−
◦

−
◦

=

Then, we will show the induction case (x = |a|−◦ z). Note that we should assume that y = v−◦w
and width v = 1 for the consistency. Also, z and w should have the same width.
The right hand side is calculated as follows.
−◦◦ w)))) (BRs (|a|−◦ z))))
flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs (v −{ definition of BRs }flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)−◦ (BRs w))))(mapr (λt. zipwith (−◦) t (top (BRs z))) (BRs |a|)−◦ (BRs z)))){ simplification }flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)−◦ (BRs w)))) ((||a|−◦ z|)−◦ (BRs z))))){ definition of left and mapr }flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v))−
◦ left (BRs w))) ((||a|−◦ z|)−◦ (BRs z)))){ split of mapc }flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)))) (||a|−◦ z|)−◦ (mapc (λt. zipwith ( ) t (left (BRs w))) (BRs z)))){ definition of map and flatten }flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)))) (||a|−◦ z|)))−◦ flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs w))) (BRs z)))
−
◦

=

−
◦

=

−
◦

=

−
◦

=

−
◦

−
◦

−
◦

=

238
The left hand size is as follows.

=

−
◦

−
◦

−
◦

−
◦

−
◦

−
◦

=

−
◦

−
◦

′′
◦ z)) (rights (|a|−◦ z)) (lefts (v −◦ w))))
flatten (unwind−◦ (zipwith3 fs (rects (|a|−′′where fs s1 r1 l2 = s1 map (the r1 ) l2 ){ definition of rects, rights, and lefts }′′flatten (unwind−◦) ||a||) (tops z))−◦ (NIL rects z))◦ (zipwith3 fs ((|||a||| map (zipwith (−((|||a||| map (zipwith (−◦) ||a||) (toprights z))−◦ (NIL rights z))◦)) (bottomlefts v) (toplefts w))−
◦ (NIL lefts w))))((lefts v gemm ( , zipwith (−{ definition of flatten, unwind , and zipwith3 , and zipwith (−◦) ||a|| = map (|a|−◦) }|fs′′ ||a|| ||a|| (the (lefts v))|−◦ flatten (reduce ( ,−◦) (map | · | (zipwith3 fs′′ (map (map (|a|−◦)) (tops z))(map (map (|a|−◦)) (toprights z)) (gemm ( , zipwith (−◦)) (bottomlefts v) (toplefts w)))))′′−◦ flatten (unwind−◦ (zipwith3 fs (rects z) (rights z) (lefts w)))

The induction hypothesis guarantees the following equation of the last half parts of the resultsof the above calculations.

−◦

flatten (map TLs (mapc (λt. zipwith ( ) t (left (BRs w))) (BRs z)))′′= flatten (unwind−◦ (zipwith3 fs (rects z) (rights z) (lefts w)))

The rest of the proof is to show the following equation.

−◦

flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)))) (||a|−◦ z|)))= |fs′′ ||a|| ||a|| (the (lefts v))|−◦) (map | · | (zipwith3 fs′′ (map (map (|a|−◦)) (tops z))◦ flatten (reduce ( ,−(map (map (|a|−◦)) (toprights z)) (gemm ( , zipwith (−◦)) (bottomlefts v) (toplefts w)))))The left hand side is calculated follows.

−◦

flatten (map TLs (mapc (λt. zipwith ( ) t (left(mapr (λt. zipwith (−◦) t (top (BRs w))) (BRs v)))) (||a|−◦ z|))){ distribution of left }flatten (map TLs (mapc (λt. zipwith ( ) t (mapr (λt. zipwith (−◦) t (left (top (BRs w))) (left (BRs v))))) (||a|−◦ z|))){ left (top (BRs w)) = |w|, and left (BRs v) = |v| }◦) t |w|) |v|)) (||a|−◦ z|)))
flatten (map TLs (mapc (λt. zipwith ( ) t (mapr (λt. zipwith (−{ simplification }flatten (map TLs (|(|a|−◦ z) (v −◦ w)|))◦ w))
TLs ((|a|−◦ z) (v −
−
◦

=

−
◦

=

=

−
◦

−
◦

=

239

Bibliography
The right hand side is as follows.

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

−
◦

−
◦

=

−
◦

−
◦

=

−
◦

−
◦

=

−
◦

−
◦

=

|fs′′ ||a|| ||a|| (the (lefts v))|−◦)) (tops z))◦) (map | · | (zipwith3 fs′′ (map (map (|a|−◦ flatten (reduce ( ,−(map (map (|a|−◦)) (toprights z)) (gemm ( , zipwith (−◦)) (bottomlefts v) (toplefts w))))){ definition of fs′′ , and simplification }||a|| map (|a| ) (the (lefts v))−◦ flatten (reduce ( ,−◦) (map | · | (zipwith3 fs′′ (map (map (|a|−◦)) (tops z))(map (map (|a|−◦)) (toprights z)) (gemm ( , zipwith (−◦)) (bottomlefts v) (toplefts w))))){ flatten ◦ reduce ( ,−◦) ◦ map | · | = reduce ( ,−◦) }||a|| map (|a| ) (the (lefts v))−◦ reduce ( ,−◦)) (tops z))◦) (zipwith3 fs′′ (map (map (|a|−(map (map (|a|−◦)) (toprights z)) (gemm ( , zipwith (−◦)) (bottomlefts v) (toplefts w))){ bottomlefts v is a singleton, and the definition of gemm }||a|| map (|a| ) (the (lefts v))−◦ reduce ( ,−◦) (zipwith3 fs′′ (map (map (|a|−◦)) (tops z))(map (map (|a|−◦)) (toprights z)) (map (zipwith (−◦)) (the (bottomlefts v)) (toplefts w)))◦) }{ definition of reduce ( ,−◦) (|||a|| map (|a| ) (the (lefts v))|
reduce ( ,−(zipwith3 fs′′ (map (map (|a|−◦)) (tops z))(map (map (|a|−◦)) (toprights z)) (map (zipwith (−◦)) (the (bottomlefts v)) (toplefts w)))){ equivalence under height v = 1 and width z = 1 }reduce ( ,−◦) (|||a|| map (|a| ) (the (toplefts v))|◦)) (toplefts z))(zipwith3 fs′′ (map (map (|a|−(map (map (|a|−◦)) (toplefts z)) (map (zipwith (−◦)) (the (toplefts v)) (toplefts w)))){ change of fs′′ to ftl′ }◦) (|||a|| map (|a| ) (the (toplefts v))|
reduce ( ,−(zipwith ftl′ (map (map (|a|−◦)) (toplefts z))(map (zipwith (−◦)) (the (toplefts v)) (toplefts w))))
ftl′ tl 1 tl 2 = tl 1 map (the tl 1 ) tl 2 ) where′right |a| = the |a|, and map (|a|−◦) = zipwith (−◦) ||a||◦) (ftl ||a|| (the (toplefts v))
reduce ( ,−(zipwith ftl (map (zipwith (−◦) ||a||) (toplefts z))(map (zipwith (−◦)) (the (toplefts v)) (toplefts w))))where ftl tl 1 tl 2 = tl 1 map (right ′ tl 1 ) tl 2 ){ definition of zipwith }◦) (zipwith ftl (|||a||| map (zipwith (−◦) ||a||) (toplefts z))
reduce ( ,−(toplefts v map (zipwith (−◦)) (the (toplefts v)) (toplefts w)))
{ definition of toplefts }reduce ( ,−◦) (toplefts ((|a|−◦ z) (v −◦ w)))
−
◦

=

−
◦

−
◦

−
◦

◦) ◦ toplefts, we have TLs ((|a| −◦ z) (v −◦Since we have shown the dilation TLs = reduce ( ,−w)) = reduce ( ,−◦) (toplefts ((|a| −◦ z) (v −◦ w))). Therefore, we have shown the induction case(x y) of the dilation of rects ′ .
The other induction case is shown similarly.

Publication Lists
International Journals
1. Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Masato Takeichi:
A Compositional Framework for Developing Parallel Programs on Two-Dimensional
Arrays.
International Journal of Parallel Programming, Vol. 35, No. 6, pp. 615–658, Springer
Netherlands, 2007.

Domestic Journals
2. 野村 芳明, 江本 健斗, 松崎 公紀, 胡 振江, 武市 正人:
木スケルトンによる XPath クエリの並列化とその評価.
コンピュータソフトウェア, Vol. 24, No. 3, pp. 51–62, 2007.

Refereed Papers (International Conferences and Workshops)
3. Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, and Masato Takeichi:
Domain-Specific Optimization Strategy for Skeleton Programs.
In 13th International Euro-Par Conference, Rennes ,France , August 28-31, 2007.
Proceedings, Lecture Notes in Computer Science 4641, pp. 705–714, Springer, 2007.
4. Kazuhiko Kakehi, Kiminori Matsuzaki, and Kento Emoto:
Efficient Parallel Tree Reductions on Distributed Memory Environments.
In Computational Science — ICCS 2007, Lecture Notes in Computer Science 4488, pp.
601–608, Springer, 2007.
5. Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, and Masato Takeichi:
Surrounding Theorem: Developing Parallel Programs for Matrix-Convolutions.
In 12th International Euro-Par Conference, Dresden, Germany, August/September 2006,
Proceedings, Lecture Notes in Computer Science 4128, pp. 605–614, Springer, 2006.
6. Kiminori Matsuzaki, Kento Emoto, Hideya Iwasaki, and Zhenjiang Hu:
A Library of Constructive Skeletons for Sequential Style of Parallel Programming.
In First International Conference on Scalable Information Systems (InfoScale 2006), May
29–June 1, 2006, Hong Kong, pp. 12, 2006.

Domestic Conference and Workshop or Non-Refereed Papers
7. Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Kiminori Matsuzaki, and Masato Takeichi:
A Generative Matrix Library in Fortress Relieves Programmers’ Headache!
日本ソフトウェア科学会第 25 回大会論文集, 筑波大学, 2009.

242
8. 江本 健斗, 松崎 公紀, 胡 振江, 武市 正人:
近傍要素を必要とするスケルトンプログラムの最適化.
第 9 回プログラミングおよびプログラミング言語ワークショップ (PPL 2007) 論文集, pp.
125–139, 2007.
9. 野村 芳明, 江本 健斗, 松崎 公紀, 胡 振江, 武市 正人:
木スケルトンによる XPath クエリの並列化とその評価.
日本ソフトウェア科学会第 22 回大会論文集, 東北大学, 2005.
10. 松崎 公紀, 明石 良樹, 江本 健斗, 岩崎 英哉, 胡 振江:
助っ人：構成的な並列スケルトンによる並列プログラミングライブラリ.
日本ソフトウェア科学会第 22 回大会論文集, 東北大学, 2005.
11. Kazuhiko Kakehi, Kiminori Matsuzaki, Akimasa Morihata, Kento Emoto,and Zhenjiang Hu:
Parallel Dynamic Programming using Data-Parallel Skeletons.
日本ソフトウェア科学会第 22 回大会論文集, 東北大学, 2005.
12. 江本 健斗, 胡 振江, 筧 一彦, 武市正人:
二次元配列上の構成的並列スケルトンの実現.
日本ソフトウェア科学会第 21 回大会論文集, 東京工業大学, 2004.

Technical Reports
13. Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Kiminori Matsuzaki, and Masato Takeichi:
Generator-based GG Fortress LibraryTechnical Report METR 2008-16, Department of Mathematical Engineering andInformation Physics, the University of Tokyo, 2008.
14. Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Kiminori Matsuzaki, and Masato Takeichi:
Generator-based GG Fortress Library—Collection of GGs and Theories—
Technical Report METR 2008-17, Department of Mathematical Engineering andInformation Physics, the University of Tokyo, 2008.
15. Kento Emoto, Kiminori Matsuzaki, Zhenjiang Hu, and Masato Takeichi:
Domain-Specific Optimization for Skeleton Programs Involving Neighbor Elements.
Technical Report METR 2007-05, Department of Mathematical Engineering andInformation Physics, the University of Tokyo, 2007.
16. Kazuhiko Kakehi, Kiminori Matsuzaki, Kento Emoto, and Zhenjiang Hu:
An Practicable Framework for Tree Reductions under Distributed Memory Environments.
Technical Report METR 2006-64, Department of Mathematical Engineering andInformation Physics, the University of Tokyo, 2006.
17. Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, and Masato Takeichi:
A Compositional Framework for Developing Parallel Programs on Two Dimensional
Arrays.
Technical Report METR2005-09, Department of Mathematical Engineering andInformation Physics, the University of Tokyo, 2005.

Presentations
18. 江本 健斗, 胡 振江, 筧 一彦, 松崎 公紀, 武市 正人:
プログラム運算に基づく最適化機能つき Fortress ライブラリ.
第 11 回プログラミングおよびプログラミング言語ワークショップ (PPL 2009), カテゴリ 3,
2009.

Bibliography

243

19. 江本 健斗, 胡 振江, 筧 一彦, 松崎 公紀, 武市 正人:
Generator-of-generators に基づく Fortress ライブラリ.
第６９回プログラミング研究会 (PRO-2008-1), 2008.
20. 江本 健斗:
助っ人：構成的な並列スケルトンによる並列プログラミングライブラリ.
第 8 回プログラミングおよびプログラミング言語ワークショップ (PPL 2006), カテゴリ 3,
2006.
21. Kento Emoto:Developing Parallel Programs for Matrix-Convolutions with Constructive Skeletons
Presentation at Informal Workshop on Skeletal Parallel Programming, the University ofTokyo, 2006.