Asynchronous Dynamic Load Balancing of Tiles

Many scienti c computations have work-intensive kernels consisting of nested loops with a regular stencil of data dependences. For such loop nests, tiling [3] is a well-known compiler optimization that can help achieve e cient parallelism. For a loop nest of depth K, tiling partitions the K-dimensional space of iterations into regular size and shape chunks of work. These tiles are allocated to di erent processors, giving parallelism and locality. In this paper, we consider the simple but common case of a doubly nested loop with a regular stencil of dependences in both dimensions. These dependences give rise to tile dependences, which require synchronization and communication of data between tiles. We address the speci c case where a tile needs data computed in the tile to its left and the tile below it to execute. Execution of tiles proceeds in a wavefront fashion: each processor will be at least one row ahead of its neighbor to the right. Rebalancing can prevent much larger imbalances that would otherwise introduce idle time. Our target architecture is a distributed-memory computer such as a Cray T3E, IBM SP, or network of workstations. Initially, each processor is allocated the same xed number of columns of tiles in block distribution. Each processor executes its allotted tiles a row at a time. If each processor has the same workload and computing capacity, then this allocation should be e cient. However, if the workloads or compute power vary, then dynamically adjusting the allocation of columns to achieve a better load balance is desirable. Varying processor loads can arise because of an evolving irregular application or because of contention in a non-dedicated system. We present the Left-Right Hando Protocol, a dynamic load balancing algorithm which does not require centralized control. This protocol can be described as a series of asynchronous "races" between pairs of contiguous processors. In alternate rounds, a processor alternates between racing against its left neighbor and its right neighbor. Each processor executes a xed percentage of its workload, then signals its neighbor it has reached its \checkpoint". If the neighbor receives this signal before reaching its own checkpoint, it may decide to hand o some of its work to the sender, assuming it determines that this will increase the performance of the both processors. To actually hand o work, the processor sends a message that includes the amount of work being transferred, and the data that the receiver will need to perform the work. A key feature of this algorithm is that the processor reaching the checkpoint rst can (without waiting for a reply) complete its row. Assuming the imbalance isn't too large, the reply will be available at the row's end, and neither processor will have any idle time. We note, however, that the protocol involves more overhead than a static schedule, since extra rounds of commuication are involved. There is an enormous amount of literature on dynamic load balancing [5]. Our scheme uses a local rescheduler; this has advantage over methods using a global scheduler, such as [1], which can require a dedicated processor and global synchronization, and may not scale well to a large number of processors. The Left-Right Hando Protocol is an \asynchronous deterministic iterative di usion" method, where the work can \di use" over time from processors with high \concentrations" of work to less busy neighbors [4]. Asynchronous methods have the potential to be faster than synchronous protocols; however, the rate of convergence and the optimal communication schedule are not theoretically known [4]. Experimental results are therefore useful in determining the practicality of asynchronous schemes. Our experiments used a simple SOR (Successive Over Relaxation) on the Cray T3E using 8 processors. Performance was measured in three scenarios: (1) Load is distributed equally among