A Data Model For Architecture Independent Parallel Programming

This paper presents a common data model for both shared and distributed parallel programming. Our model aims at parallel application programming, especially in th e field of data parallelism. We introduce coordinators shared objects which automatically perform synchronization in shared memory programming and remote communication in distributed programming. Programming with coordinators allows a static definition of dynamic access properties (shared access patterns) in a declarative manner. Thus, there is no need for explicit use of synchronization or communication primitiv es in order to get well defined and efficient parallel programs. In contrast, declarativity allows to think in abstract coor dination categories. Our paradigm is named Declarative Imperative Parallel Programming due to this fact and due to our commitment to imperative applications, mainly from the field of scientific computing, image processing, simulation and optimization. 1. The DIPP programming model Ideally, explicit parallel programming should be uniform for shared and distributed platforms. One way to achieve this is to give the programmer the illusion of shared memory even on distributed platforms[18]. While we think that this might be exactly what a programmer wants we must acknowledge that this goal is not reached yet. The performance issue implies some sort of discrimination of local from non-local data for distributed platforms. We believe that the process model can be uniform without any impact on performance. We propose such a model (a workpool model) in [13]. The aim of this paper is to show that even the data model can benefit from an uniform approach allowing architecture invariant programmin g without too much loss of performance. The relationship between our data model and the workpool is described in [14]. The main property of the approach presented in this paper is a paradigm which makes explicit synchronization or remote communication obsolete without performance degradation1. We introducecoordinators, a means for declaratively specifying the dynamic access properties of objects to be used by more than one process. The access properties directly correspond to algorithmic patterns typically found i data parallel algorithms such as fan-in or fan-out. In the context of message passing we have the related term broadcast. These all are specific cases of access patterns which can be defined by coordinators. The automatic coordination behavior of a system that implements coordinators can be derived from the specified patterns. Since coordinators are a declarative means to be used in imperative programming the programming model is named Declarative Imperative Parallel Programming. Imperativity is one of the properties which make coordinators different from other implicit synchronization paradigms such as futures [10]. See the discussion of related work for details. Coordinators are objects that may be accessed by more than one process in a coordinated fashion. In shared memory programming they might be named selfsynchronized objects. In distributed programming the term self-distributing might be in order. Accesses to coordinators may trigger synchronization or remote communication actions. We discriminate writing accesses (producers) from reading accesses (consumers). Note, that we do not support general producer-consumer aspects like buffering. In contras t, coordinators are a means for synchronized control allowing a tight coupling between producer and consumer. We 1Notice that we are concerned with application programming; we do not claim that our model does away with explicit synchroniza tion or coordination for parallel system programming such as operati ng systems or databases. specify access properties in a declarative manner: Access patterns are declared separately for reading and writing. The parallel semantics is defined by switching between a writableand areadablestate of a coordinator. The patterns statically describe the conditions which trigger the state switches. These conditions are defined in terms of “who reads or writes an object how many times”. Inappropriate (premature) accesses are delayed until the state switches. This behavior is implemented by implicit synchronization or communication, depending on the underlying architecture. 1.1. Dynamic behavior of coordinators The basic assumption about accesses to coordinators is that writing accesses are distinguished from reading accesses. In imperative programs, switches between writable and readabletypically occur repeatedly inside loops. In functional programs there is exactly one switch from writable to readable. 2 In the following, we informally describe the dynamic behavior of coordinators. Two state semantics Coordinators exhibit a two state semantics: A coordinator is initially in thewritablestate. After construction of the first value it switches to the readable state. After the last read access the coordinator switches t o thewritable state again, thus allowing multiple updates of objects. Depending on the access patterns defined for each state (see section 1.2), the switches may be triggered by dif ferent actions. Blocking If a coordinator is accessed and its state does not correspond to the access kind (i.e. a read access while the coordinator is writable and vice versa) the access is automatically blocked. Blocking lasts at least until the stat e has switched (e.g. by completion of some other accesses). By defining the switch conditions between the two states, the access patterns also determine blocking conditions. Computational steps Our focus is on iteratively writing and reading objects inside some loop or recursion. All such activities within one iteration are referred to as a computational step. More accurately, we define a computational stepwith respect to a coordinator as (dynamically) starting in the writable state and ending when this state is reached again, i.e. a computational step consists of a write phase and aread phase . 3 2Readers familiar withfuturesor I-structuresmight see some relationship to these. 3In the functional world, the whole lifetime of a variable is a computational step. The semantics of coordinators is defined by their behavior during one computational step. In general, different co ordinators will carry out different computational steps. 1.2. De ning access patterns Every coordinator reveals a specific access pattern with respect to both phases of a computational step. The access patterns must be specified by the user separately for both phases. Access patterns correspond to algorithmic pat terns as they typically occur in “data-oriented” parallel a lgorithms (e.g. in numerical computation). We have defined the following patterns: each(<process group >,<no of accesses >) each(<process group >) arbitrary(<no of accesses >) arbitrary() The main patternseach and arbitrary differ in the mode of cooperation between the involved processes while their shapes denote the events that trigger a state change. Theeach pattern is used for a homogeneous group of processes <process group >, when the overall task within the write or read phase of a computational step can be defined by specifying the task of every single process within the phase. The intended standard way to specify such a task is to declare how many times the process will access the coordinator within the phase. To do so the each pattern is used with a<no of accesses > specification ( <no of accesses > will be 1 in many cases). This pattern fits for most algorithms that work on regular data (like arrays) in a regular fashion. If the number of accesses cannot be given, each process has to state explicitly that its task within a phase is finished by invoking the coordinator method procReady(<phase>). This method is parameterized with the phase, since it must be synchronized with concurrent accesses. Thus, a call to procReady is a further potential blocking situation. The state change is triggered as soon as each process has finished its task. In order to get a proper definition of this switch condition we require an invariable line-up of the pro cess group (except for an initial phase, see [13]). In contrast, thearbitrary pattern is used if an arbitrary, possibly varying number of possibly inhomogeneous processes share the coordinator. Within this cooperation mode, we can provide only global switch conditions: In the arbitrary pattern a<no of accesses > specification refers to the overall number of accesses, i.e. the state of the coordinator switches after exactly <no of accesses > accesses, no matter which processes were involved. Again, if the number of accesses cannot be given the state switch has to be triggered explicitly. In contrast to theeach pattern this is done by only one process, using the coordinator method switch(<phase>). It can be concluded that this pattern is appropriate e.g. for a farm work model where a centralized master controls the work of several slave processes. 1.3. A short example We illustrate the use of coordinators in a small example, using aC++ like notation enhanced with simple syntactic extensions. Our example implements a producer-consumer scenario where an integer variable sharedInt is repeatedly written by a single producer and subsequently read by nProcs different consumers. We ensure that each value written by the producer is read exactly once by each consumer, by simply definingsharedInt as a coordinator: