Programming Models for Parallel Systems : The Programmer ’ s Perspective 1 st Author

Until now the mainstream software development community has not had to deal with the challenges of concurrent or parallel programming. Concurrent programming has been confined to specialized domains with closed world assumptions e.g. operating system kernels. Parallel programming has been successfully performed in very specific domain e.g. scientific computing using highly skilled developers often on special parallel systems (e.g. supercomputers). Even more specialized are FPGA-based systems which provide parallel hardware implementations of problems which can outperform supercomputers. However, the highly specialized skills one needs to produce these systems makes them inaccessible to the mainstream developer or user of computationally intensive systems. All this is about to change as we approach an era where single core performance has leveled off and processor vendors have started to move to a model where the number of cores available in a single die will double with each silicon generation [1]. What are we going to do with all those processors? How will we program them? Will we have to learn how to write concurrent and parallel programs? Or can we produce some special technology to take our sequential programs and automatically parallelize them? Are there better ways of using all that silicon other than stamping out many instances of the same core and connecting them through a series of caches to the same shared memory? We are now approaching an era where mainstream programmers may face some of the same challenges that have faced the parallel hardware and reconfigurable computing communities. Every challenge can also lead to an opportunity. Today we have a sad state of affairs at the point where hardware meets software. Hardware and software are developed using quite different models of computation. Systems that comprise a mixture of hardware and software are difficult to design because it is hard to realte C components to VHDL components. The author believes that the change in processor architectures will necessarily force a change in the way we write software and the underlying models that we use to understand our software. This is an opportunity to carefully design models which make it easier to relate software to hardware and to even convert software descriptions into hardware descriptions. One important reason for being able to convert a software implementation of a calculation into a hardware implementation is the author’s belief that future many-core architectures will be very heterogeneous. They will comprise not only many processors of different kinds but also of other components that can compute e.g. reconfigurable data-paths that are the evolution of today’s GPUs and reconfigurable fabrics that are the evolution of today’s FPGAs. These devices may no longer have a single coherent memory and instead provide local memory to tens or hundreds of processors which are connected together via an on-chip network. To allow applications to run effectively on mixtures of such “Metropolis” architectures (see Figure 1) will require the ability to migrate calculations from one kind of computation element to another e.g. from software to hardware. Figure 1. A potential many-core architecture. The problem of how to model, write, integrate, test and verify such systems a daunting task but we have some good existing work to guide us. We absolutely need to think in terms of different models of computation for different parts of our application and then we need a way to compose or relate these models of computation. The Ptolemy II [5] system provides such a framework and has been successfully used to model embedded systems and it has also been used to generate hardware from certain models of computation. A successful future parallel systems programming model is likely to borrow some of the key ideas from Ptolemy II to help structure and compose the elements of heterogeneous parallel systems. For programming such architectures it is essential to allow the programmer to express relevant aspects of the parallel computation in a productive manner without saying too much about a particular parallel implementation architecture and here we have the example of work like the Chapel programming language [6] developed for high performance computing to help guide us in the programming language space. Chapel, along with languages like Fortress [10] presents a model that abstracts the number and kind of computational resources available to the programmer. There has been much work on compiling C or C-like languages to hardware. With a few notable exceptions, there has been limited progress in this area. The author believes that starting from C one has already introduced so much unnecessary sequentiality that it is often impossible to recover the potential parallelism in the original algorithm. Furthermore, C does not have explicit constructs for expressing concurrency or parallelism and this makes it even more difficult to devise automatic parallelization flows. Some have suggested compiling multi-threaded code into hardware or even explicitly implementing the notion of hardware threads. However, programs written explicitly in terms of threads and locks are likely to be at the wrong level of abstraction for effective compilation into a variety of parallel computing elements especially FPGA-like fabrics. Threads and locks are systems programming abstractions that have proved effective for their original use but it is not clear that they are the appropriate abstractions for writing concurrent or parallel applications (hardware, software or a mixture) [16][15]. This programmer’s perspective on how to write for parallel systems yearns for higher level constructs which capture the salient parallel operations and leave enough details unsaid to allow efficient compilation to many different kinds of parallel computing elements. Even modestly raising the concurrency abstraction can provide a better model for writing parallel systems and a better starting point for hardware synthesis tools. Here are some examples of such higher level concurrency abstractions:  Futures. This technique implicitly introduces parallelism by identifying calculations whose values will be needed at a later stage and may be scheduled for parallel execution. There has been much research into compiling future based descriptions into software. But can future based descriptions also be synthesized into efficient hardware?  Cilk. The Cilk programming language [2] adds three keywords to C which allows many interesting kinds of parallelism to expressed without the explicit mention of system level threads. These extensions can always be removed from a Cilk program to yield a valid C program. Very efficient compilation and run-time technology has been produced for Cilk. This may be a much viable starting point for a C to gates style flow than conventional methods.  Single assignment variables. These are variables which can only be assigned once. If a parallel activity tries to read from an unassigned value it is blocked. This simple mechanism often provides a nice way to synchronize parallel activities and is effectively implemented in systems that have memory with full/empty bits. Can parallel systems described in terms of single assignment variables be translated into effective circuits which communicate and synchronize using appropriate hardware structures e.g. single element buffers?  Software transaction memory. This technique [13] takes the notion of a transaction from the database world and applies it to a shared memory concurrent programming context. It allows multiple parallel activities to operate on shared information without taking explicit locks. This approach eliminates many kids of deadlocks and may also give rise to optimistic concurrency. Can this model be used to compose hardware and software blocks in a parallel system with shared memory? Preliminary work has shown that some other kinds of concurrency abstractions can be modeled using STM e.g. join patterns [18].  Join patterns. Several research languages have experimented with the notion of using join patterns [11][12] for synchronization between parallel activities. An example is the COmega [3] language which nicely integrates joins into the object model of C# (similar work has been done with Java [14] and OCaml [8]) and they have also been implemented as a library [7]. At least on research group has investigated compiling join pattern based descriptions into hardware and this approach merits further investigation as a way to help synchronize parallel activities in hardware and software. Join based descriptions raise the level of abstraction at which asynchronous message passing programming is performed and join patterns can effectively encode many other kinds of concurrency abstractions.  Data parallelism. The approaches presented so far belong to the class of control oriented concurrency. By far the most effective way of producing parallel systems that map well to multiple processors or FPGA-like hardware is the use of data-parallel descriptions. These descriptions can often be compiled into very effective pipelined parallel hardware on reconfigurable platforms. Furthermore, data parallel programming is a much easier “sequential” model for the programmer than a thread based model which results in non-deterministic behavior. Challenges remain for the design of parallel systems built in a composable manner and there needs to be more work in the area of nested data parallelism.  Functional programming. Programming languages that are inherently more amenable to parallelism are likely to play a more important role in the future, especially for data-parallel programming e.g. concurrent Haskell [17]. These are just a few of the techniques and approaches we can take to help us develop methodologies for programming parallel systems whether they are software, hardware or ambiguous. Other challenges aris