Parallel Sorting by Overpartitioning

The approach of parallel sorting by Overpartitioning (PSOP) limits the communication cost by moving each element between the processors at most once, and ensures good load balancing (even distribution of data among the processors). Introduction In a typical sorting algorithm each processor contains a portion of the list of n elements to be sorted. In a merge based sort, each processor sorts the portion it holds initially by some sequential sorting algorithm, and then all sorted portions are exchanged among all the processors and merged. In a quicksort based approach the unsorted list is partitioned into a number of progressively smaller sublists by means of selected pivots. Then the processors employ a sequential sorting algorithm to sort the sublists they contain. The merging in merge based approach and the partitioning in the quicksort based approach can be achieved either in one step or many steps.Thus parallel algorithms can be classified as single-step or multi-step. Single-step algorithms incur low communication cost, because they move each element at most once, however, single-step algorithms often have poor l ad balancing because it is difficult to derive sublists of nearly equal sizes in absence of global information and there is no way to rearrange the sublists once they are derived. On the other hand, multi-step algorithms incur higher communication cost because elements may be moved several times between processors. So the main problem with single step algorithms is poor load balancing. The following approach to single-step parallel sorting provides good load balancing. Overview of Parallel Sorting by Overpartitioning (PSOP) approach Given the list of n unsorted data items, each processor initially gets a list of n/p elements. The list in each processor is then divided into k times as many sublists as there are processors by selecting (pk-1) pivots (p << n). i.e., number of sublists = k * p; k is called the Overpartitioning ratio, and the reason for selecting the number of sublists to be a multiple of p is to ensure that all processors have a sublist to sort during the local sorting phase. The sorting is accomplished in the following four phases. Phase 1: Given the unsorted list l of n elements and p processors, equal number (n/p) of elements are distributed to each processor. Consider n=39,p=3, we have p1 : 10 21 1 2 26 29 7 33 9 20 3 16 41 p2 : 6 40 59 5 14 37 18 8 27 15 32 23 11 p3 : 11 44 12 3 19 28 34 31 42 17 48 0 9 Phase 2: A sample of pks (ks from each processor) are randomly picked. ( s is called the Oversampling ratio, which is one of the factors deciding on the number of samples to be picked from each processor), in this case s=1), i.e., without Oversampling. sample of pks candidates : 10 26 40 14 44 12 Sort the sample sorted sample : 10 12 14 26 40 44 Now select (pk-1) elements from the sample by taking elements from the sorted sample at positions s,2s,. ,(pk-1)s. (pk-1) pivots are : 10 12 14 26 40 Broadcast all (pk-1) pivots to all processors. Phase 3: Starting from the middle pivot, partition each list into left and right half. Then recursively partition the left and the right half using the left and right pivots respectively. p1 : 1 2 7 9 3 10 21 16 20 26 29 33 41 p2 : 6 5 8 11 14 18 15 23 37 27 32 40 59 p3 : 3 0 11 12 19 17 28 34 31 39 44 42 48 Sublists S1 S2 S3 S4 S5 S6 Phase 4: Compute the sublists sizes and their final positions in the sorted list and build a task queue. When a processor is ready, it gets a task from the queue and sorts the sublist of the task until he q eue is empty. sublists : S1 S2 S3 S4 S5 S6 sizes : 10 3 1 9 10 6 final position : 0 10 13 14 23 33 task queue : T(S1), T(S5), T(S4), T(S6), T(S2), T(S3) Size of a sublist is computed as follows p |Sj| = ∑ |l ij| i=1 Now compute starting position of sublist S j in the final sorted array, given by j-1 σj = 1 + ∑ |Sh| h=1 Note: The task queue is built with tasks ordered from the largest sublist to the smallest. Each processor repeatedly takes one task, T(Sj) at a time from the queue. It processes the task by a) copying the p parts of the sublist, (l ij .,lpj), into the final array at positions σj to σj + |Sj|-1 and b)applying a sequential sort to the elements in that range. This process continues until the task queue is empty. Since the tasks in the queue are in decreasing order of sublist size, in phase 4 every processor gets larger tasks at the beginning and smaller tasks towards the end. Overpartitioning leads to a sufficient number of tasks of varying sizes such that PSOP achieves good load balancing. Which sublist is routed to which processor depends upon the starting position of the sublist.If starting position is between 0 and n/p-1, then that sublist is routed to processor 1, if it is between n/p and 2*n/p 1, it is routed to processor 2, etc. ex: p1 takes S1 from T(S1),sorts it and routes it to p1,because the starting position of sublist S1 is 0. Similarly p2 takes S5, sorts it and routes it to p2 because it’s starting position is 23. Continuing this way, we get processor p1 p2 p3 size 13 20 6 We note that the processors do not have same number of elements. What if it is required to store equal number (n/p) of elements in each processor. The following method accomplishes this. Compute Partial Sum (P.S.) and Total Sum (T.S.) for each processor. processor p1 p2 p3 size 13 20 6 P.S. 13 23 39 T.S. 39 39 39 Each processor is required to have T.S. /p elements. To achieve this, All processors compute the number of extra elements they have, then they route their last (size T.S./p) elements the neighbouring processor. i.e., processor i routes it’s extra elements to processor i+1. In a single route all processors will have exactly n/p elements. Now our goal is to reduce load imbalance as much as possible. Before looking at how the values of s and k can reduce load imbalance, let us define Sublist expansion and Load expansion. Sublist expansion This is defined as the ratio of the maximum sublist size to the mean sublist size maxj=1,q |Sj| ----------------q = number of sublists q ∑|Sj|/q j=1 A large sublist expansion indicates the size of the largest sublist is very large when compared to the mean sublist size. It can be shown that load imbalance is proportional to sublist expansion.