Scalable data abstractions for distributed parallel computations

The ability to express a program as a hierarchical composition of parts is an essential tool in managing the complexity of software and a key abstraction this provides is to separate the representation of data from the computation. Many current parallel programming models use a shared memory model to provide data abstraction but this doesn't scale well with large numbers of cores due to non-determinism and access latency. This paper proposes a simple programming model that allows scalable parallel programs to be expressed with distributed representations of data and it provides the programmer with the flexibility to employ shared or distribu ted styles of data-parallelism where applicable. It is capable of an efficient implementation, and with the provision of a small s et of primitive capabilities in the hardware, it can be compiled to operate directly on the hardware, in the same way stack-based allocation operates for subroutines in sequential machines. I. I NTRODUCTION When developing a program of any complexity, the ability to express it in terms of a simpler set of components is essential. A component presents a simple interface that allows its implementation to be considered independently, and when combined with other components, the internal details can be ignored and its functionality treated in an abstract way. This allows a program to be constructed using modules, ranging from small functions to libraries, and for any component to be substituted with another that adheres to the same interface. The importance of abstraction as a tool in computer programming was recognised by Turing in the 1940s (1) and was formalised in the 1970s by the structured programming methodology (2). This aimed to improve the quality of programs and productivity of pro- grammers through judicious use of hierarchical structuring and subroutines. These principles have been foundational for modern sequential programming languages. A key issue with composability is separating the rep- resentation of data from the structure of a computation. Mainstream CPU and general-purpose GPU (GPGPU) par- allel programming models are based on a shared memory model, where data are globally accessible. This is the form of parallel random access machine (PRAM) (3) and the related bulk-synchronous parallel (BSP) (4) model. Shared memory parallelism allows sequential approaches to data abstraction and conventional data structures to be employed, but it does not scale well with large numbers of cores. Access latency can vary significantly and unpredictably due to the physical distribution of data across a machine. This makes it difficult to exploit locality, which is essenti al for scaling a computation, and poses problems for barriers which are delayed by the slowest participant. Additionally, when accesses are made to shared data they can incur latency from collisions, and when they are updating it, behaviour can become non-deterministic. There are a number of issues related to the implementation of a shared memory system that pose further problems for this type of data abstraction. Mainstream parallel process ors take the form of symmetric multi-processors (SMPs) and these have brought about a number of parallel programming approaches such as the Cilk (6) language, OpenMP (7) and Intel's Threaded Building Blocks (TBB) (8). These employ a multi-threaded execution model where a number of threads are managed by a scheduler, but problems can arise with pro- grams that combine parallel components. Performance can be affected significantly by threads competing for executio n, causing unnecessary context-switches, and idling within a component due to a load imbalance, causing under utilisa- tion. The effects of this are dependent on combinations of program components and result in unpredictable execution time, exacerbating non-deterministic behaviour. OpenCL (9) is emerging into the mainstream and is designed to support the programming of heterogeneous systems. These are typ- ically comprised of CPUs and GPGPUs. It uses a shared memory model but exposes distinct address spaces and in order to compose components operating in different ones, variables must be explicitly transferred between them (11). Parallelism is now the primary means of sustaining growth in computational performance (12) and the shared memory model will continue to be useful. However, it looks certain that future systems will involve large numbers of processors and it will not be effective in delivering performance on them. Therefore, it is necessary that parallel programming models, as well as supporting shared memory approaches, also support composable representations of distributed data. This paper proposes a simple distributed programming model that builds on the approach of the occam program-