ADaPT : An Event-Passing Protocol for Reducing Delivery Costs in Scatter-Gather Parallel Processes

The scatter-gather event pattern is commonly used in many high-performance codes to deliver data subsets for use in data-parallel processes and return the results. There are a number of collective algorithms and multiple collective calls in the ubiquitous Message Passing Interface standards which perform aspects of gathering. Programmers have traditionally been left to their own devices, however, to perform complex scatterings of data subsets to processors executing the dataparallel computation. We have developed ADaPT, an Adaptive Data-parallel Publication/Subscription Transport protocol, to reduce the time lost to excessive synchronization with the unicast event-passing commands often employed to scatter data. I. I NTRODUCTION In recent years, as the performance-to-cost ratio of consumer hardware has continued to increase, computational clusters consisting of fast networks and commodity hardware have become a common sight in research laboratory settings. Unlike much of the software developed for other platforms bearing the monikersupercomputer , parallel computations executed on these clusters do not normally rely on one-off parallel languages or compilers. Developers of these systems tend to extend commercially available off-the-shelf approaches to the communications protocols they employ to facilitate parallel code development. The Message Passing Interface [2], or MPI, is a very common event delivery system used in computational clusters with distributed, or disjoint, memory. In specifying MPI, its contributors intended to facilitate platform independent highperformance parallel codes with a number of point-to-point and collective communications commands. While MPI has gained significant acceptance in the scientific computing and high-performance software communities, multiple inefficiencies have shown themselves, including potentially excessive synchronization of processes in collective communications and lack of support for patterned communications [12]. One prevalent parallel computing paradigm that would benefit from increased support for patterned communications is data-parallel computing. In this paradigm, a large data set is partitioned and distributed, or scattered, between a number of processes performing nearly-identical or identical computations. A single process often initiates the parallel computation and serves as the collection point for the gathering of results. When the optimal aggregation of data results in a number of subsets substantially larger than the number of physical cluster nodes, a sequencing of messages containing the subsets develops and may be exploited. In order to reduce the delivery costs associated with the repeated delivery of datasets to processes in a multi-part scatter– when the number of datasets to be processed is much larger than the number of processes–we have introduced ADaPT, an Adaptive Data-parallel Publication/Subscription T ransport protocol. ADaPT, a protocol based on publish/subscribe, is a two-phase protocol in which processes capable of computing the parallel computation subscribe to the process that serves as the distributor of datasets. In this paper, we will discuss a computational task faced by the Laboratory for Neural Dynamics which can be made parallel using a data-parallel process, the limitations of the MPI standard to perform a multipart scattering of data, provide an introduction to ADaPT, and discuss further applications of ADaPT to high-performance parallel codes. II. T HE DYNAMIC SYNAPSE NEURAL NETWORK The Laboratory for Neural Dynamics is a computational science section of the NSF Engineering Research Center for Biomimetic Microelectronic Systems which combines computational electrophysiology, engineering and other disciplines to develop neural prosthetics. The Laboratory of Neural Dynamics integrates empirically-measured, realistic, and biologicallyinspired synapse functional models into artificial neural networks for the purpose of temporal signals processing. The Dynamic Synapse Neural Network, or DSNN, is a biogically-inspired artificial neural network which the Laboratory uses in natural language and general acoustics processing. First proposed by Berger and Liaw [6], [7], DSNNs offer a remarkable amount of robustness to noisy signal. A key to the functionality of the DSNN model is the balance between a small computational footprint of each biologically-accurate preand post-synaptic function and the emerging nonlinearities of network systems composed using these functions. Similarly to other artificial neural networks implemented in software, each DSNN computation relies on a number of parameters which must be adjusted in order for the network o be properly weighted, or optimized, to recognize input appropriately-similar to the data used in a training process. The search space consisting of a number of such parameters and their respective variability is far too large for “brute force” methodologies, and so we have explored a number of informed and directed search algorithms. Currently, the Laboratory optimizes our DSNNs using a genetic algorithm [3]. This search algorithm borrows from its natural analog, producing a populationof parameter sets, or genomes , evaluating the classification abilities of the networks built using these parameter sets, and then evolving the fittest parameter sets (i.e. those that produce the best classifications on the training set of inputs) into successive generations in an iterative fashion. Because of the large variability of signals which the trained DSNN must correctly classify, our training sets are large and classification of the complete set is very costly. Additionally, the interactions of parameters and classification are not completely understood, so many of the genomes utatedand cross-hybridizedfrom fit members of previous generations are completely unacceptable. Since unacceptability can be determined much more quickly than fitness, each parameter set undergoes testing on a directed subset of the full training set prior to undergoing a complete evaluation of fitness. This process can be described as two-phase evaluation. III. MPI I MPLEMENTATION The MPI standard [2] has made great effort to simplify the task of gathering a solution from disparate processes with both thegather and reduce commands. Because both the aggregation of the dataset to be processed and the mapping of parallel processes are hardwareand computation-dependent, the programmer of high-performance codes is left the decision of how to best parallelize their computation. Though MPI provides a matchingscatter command, its generic, naive approach to the mapping of datasets to processes addresses neither hardware not computation heterogeneity. In Figure 1, a case in which MPI’scatter command does not yield an optimized time to solution for the training of a DSNN is illustrated. As pointed to by Skjellum, et. al. [11], many implementations of the MPI standards rely on compositions of synchronized point-to-point communications in order to build collective operations such as scatter and gather . Since the genetic algorithms developed in the Laboratory of Neural Dynamics employ populations of genomes substantially larger than the numbers of physical nodes in the computational clusters with which we train our networks, the need for a multipart scatter arises. Reliance on the MPIscatter yields the process flow seen in Figure 1 where a subset of the genomes are scattered synchronously to each process, evaluated, and gathered again at the main process before the second and third subsets of genomes in the population can be evaluated. As we are using the two-phase evaluation as described in the previous section, a number of processes complete substantially earlier than others. Due to the synchronous nature of the MPI gather command, these processes remain idle until all complete, forming an P ro c 1