Parallel Strategies for Task Scheduling Algorithms Using PVM ? Stella C.S. Porto y Depto . de Eng . de Telecomunica?c ~ oes Universidade Federal Fluminense Rua Passos da P?atria 156 Niter?oi 24210 , RJ e-mail : stella@inf.puc-rio.br Celso C. Ribeiro z Departamento de Inform?atica Pontif ?? cia Universidade Cat?olica Rua Marqu ^ es de S ~ ao Vicente 225 Rio de Janeiro 22453-970 , RJ e-mail : celso@inf.puc-rio.br PUCRioInf-MCC06 / 95 April 1995 Abstract This paper presents parallelization strategies of a tabu search algorithm for the task scheduling problem on heterogeneous processors under task precedence constraints . Parallelization relies exclusively on the decomposition of the solution space exploration . Four different parallel strategies are proposed and implemented on an asynchronous parallel machine under PVM : the master-slave model , with two different schemes for improved load balancing , and the single-program-multiple-data model , with single-token and multiple-token message passing schemes . The comparative analysis of these distinct versions shows that the tabu search approach for this problem is so ideally adapted to the parallelization of the neighborhood search , that efficiency values are almost always close to one for problems over a certain size . ? This work was done while the authors were visiting the Laboratoire de Mod?elisation et Calcul at the Universit?e Joseph Fourrier in Grenoble , France . yWork of this author was sponsored by the Brazilian Ministry of Education , through a CAPES scolarship in the framework of the PDEE program . zWork of this author was sponsored by the CAPES/COFECUB Brazil-France agreement , in the framework of project 128/91 . Resumo Este artigo apresenta estrat?egias de paraleliza?c ~ ao de um algoritmo de busca tabu para o problema de escalonamento de tarefas em processadores heterog ^ eneos sob restri?c ~ oes de preced ^ encia . A paraleliza?c ~ ao se baseia exclusivamente na decomposi?c ~ ao da explora?c ~ ao do espa?co de solu?c ~ oes . Quatro estrat?egias paralelas diferentes s ~ ao propostas e implementadas sobre em m?aquina paralela ass ?? ncrona sob PVM : o modelo mestre-escravo , com dois esquemas diferentes para um melhor balanceamento de carga , e o modelo \ singleprogram-multiple-data '' , com esquemas do tipo \ single-token '' e \ multiple-token '' para a troca de mensagens . Uma an?alise comparativa destas vers ~ oes distintas demonstra que o enfoque de busca tabu para este problema ? e de tal forma bem adaptado ? a paraleliza?c ~ ao da busca na vizinhan?ca , que a efici ^ encia resultante ? e quase sempre pr?oxima de um para problemas acima de um certo tamanho . Palavras-chave : Escalonamento de tarefas , busca tabu , algoritmos paralelos , mestreescravo , SPMD . 1 Introduction When parallel application programs are executed on MIMD machines , the parallel portion of the application can be speeded up according to the number of processors allocated to it . In a homogeneous architecture , where all processors are identical , the sequential portion of the application will have to be executed in one of the processors , considerably degrading the execution time . A faster processor tightly coupled to smaller ones , responsible for executing the serial portion of the parallel application , may lead to higher performance -LSB- 13 -RSB- . In a homogeneous multiprocessor environment , one has to be able to determine the optimum number of processors to be allocated to an application -LRB- processor allocation -RRB- , as well as which tasks will be assigned to each processor -LRB- processor assignment -RRB- . In a heterogeneous setting , one has to determine not only how many , but also which processors should be allocated to an application , as well as which processors are going to be assigned to each task . The task scheduling problem in a heterogeneous multiprocessor environment with applications represented by task precedence graphs was first considered by Porto and Menasc?e in -LSB- 14 , 15 -RSB- . The focus of this work was on the processor assignment problem , assuming that processor allocation had already been performed . Greedy algorithms for processor assignment of parallel applications modeled by task precedence graphs in heterogeneous multiprocessor architectures have been proposed and compared . More recently , Porto and Ribeiro -LSB- 16 -RSB- applied the tabu search metaheuristic to the static task scheduling problem under precedence constraints in a heterogeneous multiprocessor environment . The results obtained by tabu search improved by approximately 25 % the makespan -LRB- i.e. , the completion time -RRB- of the parallel applications , with respect to the schedule generated by the best greedy algorithm . Tabu search is a local search adaptive procedure for solving combinatorial optimization problems , which guides a hill-descending heuristic to continue exploration without becoming confounded by an absence of improving moves , and without falling back into local optima from which it previously emerged -LSB- 16 -RSB- . In the case of the task scheduling problem considered in this work , where the precedence constraints determine a cost function whose evaluation takes O -LRB- n 2 -RRB- time complexity -LRB- where n stands for the number of tasks to be scheduled -RRB- , the computational times observed for tabu search may be very large . In this sense , stands out the idea of a parallel implementation of this algorithm , considering the promising results focusing other combinatorial problems already reported in the literature -LSB- 1 , 2 , 4 , 5 , 6 , 7 -RSB- . In Section 2 we present the formulation of the task scheduling problem . In Section 3 we briefly describe the tabu search approach for this problem . In Section 4 we describe the proposed parallel strategies . In Section 5 we present the test framework used for the computational experiments , followed by the numerical results and a comparative analysis of the different strategies . In Section 6 some concluding remarks are presented . 2 Problem Formulation According to the same formulation used in -LSB- 14 , 16 -RSB- , a parallel application ? with a set of n tasks T = ft1 ; : : : ; tng and a heterogeneous multiprocessor system composed by a set of m interconnected processors P = fp1 ; : : : ; pmg can be represented by a task precedence graph G -LRB- ? -RRB- and an n ? m matrix ? , where ? kj = ? -LRB- tk ; pj -RRB- is the execution time of task tk 2 T at processor pj 2 P . Given a solution s for the scheduling problem , a processor assignment function is defined as the mapping As : T ! P . A task tk 2 T is said to be assigned to processor pj 2 P in solution s if As -LRB- tk -RRB- = pj . The task scheduling problem can then be formulated as the search for an optimal mapping of the set of tasks onto that of the processors , in terms of the makespan c -LRB- s -RRB- of the parallel application , i.e. , the completion time of the last task being executed . At the end of the scheduling process , each processor ends up with an ordered list of tasks that will run on it as soon as they become executable . A feasible solution s is then characterized by a full assignment of processors to tasks . At any time instant , a task tk 2 T may be in one of the following four states : non-executable , if at least one of its predecessor tasks was not yet executed ; executable , if all its predecessor tasks were already executed but its own execution has not yet started ; executing , if it is being executed -LRB- i.e. , it is active -RRB- ; or executed , if it has already completed its execution in processor As -LRB- tk -RRB- . A processor pj 2 P may be in one of the following states at a given time : free , if there is no active task allocated to it ; or busy , if there is one active task allocated to it . The maximum completion time -LRB- makespan -RRB- of a parallel application may be computed by a labeling technique -LSB- 16 -RSB- . Algorithm makespan in Figure 1 describes the computation of the makespan of a parallel application in O -LRB- n 2 -RRB- time complexity . At the end of this procedure , c -LRB- s -RRB- = clock is the cost of the current solution , i.e. , the makespan of the parallel application given the task schedule associated with solution s. algorithm makespan begin Let s = -LRB- As -LRB- t1 -RRB- ; : : : ; As -LRB- tn -RRB- -RRB- be a feasible solution for the scheduling problem , i.e. , for every k = 1 ; : : : ; n , As -LRB- tk -RRB- = pj for some pj 2 P clock state -LRB- pj -RRB- free 8pj 2 P start -LRB- tk -RRB- ; finish -LRB- tk -RRB- 8tk 2 T while -LRB- 9tk 2 T j state -LRB- tk -RRB- < > executed -RRB- do begin for -LRB- each tk 2 T j state -LRB- tk -RRB- = executable -RRB- do if -LRB- state -LRB- As -LRB- tk -RRB- -RRB- = free -RRB- then begin state -LRB- tk -RRB- executing state -LRB- As -LRB- tk -RRB- -RRB- busy start -LRB- tk -RRB- clock finish -LRB- tk -RRB- start -LRB- tk -RRB- + ? -LRB- tk ; As -LRB- tk -RRB- -RRB- end Let i be such that finish -LRB- ti -RRB- = mintk2T j state -LRB- tk -RRB- = executingffinish -LRB- tk -RRB- g clock finish -LRB- ti -RRB- for -LRB- each tk 2 T j state -LRB- tk -RRB- = executing and finish -LRB- tk -RRB- = clock -RRB- do begin state -LRB- tk -RRB- executed state -LRB- As -LRB- tk -RRB- -RRB- free end end c -LRB- s -RRB- clock end makespan Figure 1 : Computation of the makespan of a given solution 3 Tabu Search Applied to the Scheduling Problem To describe the tabu search metaheuristic , we first consider a general combinatorial optimization problem -LRB- P -RRB- formulated as to minimize c -LRB- s -RRB- subject to s 2 S ; where S is a discrete set of feasible solutions . Local search approaches for solving problem -LRB- P -RRB- are based on search procedures in the solution space S starting from an initial solution s0 2 S . At each iteration , a heuristic is used to obtain a new solution s in the neighborhood N -LRB- s -RRB- of the current solution s , through slight changes in s . A move is an atomic change which transforms the current solution , s , into one of its neighbors , say ? s. Thus , movevalue = c -LRB- ? s -RRB- ? c -LRB- s -RRB- is the difference between the value of the cost function after the move , c -LRB- ? s -RRB- , and the value of the cost function before the move , c -LRB- s -RRB- . Every feasible solution ? s 2 N -LRB- s -RRB- is evaluated according to the cost function c -LRB- :-RRB- , which is eventually optimized . The current solution moves smoothly towards better neighbor solutions , enhancing the best obtained solution s ? . The basic local search approach corresponds to the so-called hill-descending algorithms , in which a monotone sequence of improving solutions is examined , until a local optimum is found . Tabu search -LSB- 8 , 9 , 10 , 11 -RSB- may be described as a higher level heuristic for solving combinatorial problems , designed to guide other hill-descending heuristics in order to escape from local optima . Thus , tabu search is an adaptive search technique that aims to intelligently explore the solution space in search of good , hopefully optimal , solutions . Broadly speaking -LSB- 3 -RSB- , two mechanisms are used to direct the search trajectory . The first is intended to avoid cycling through the use of short term memories -LRB- tabu lists -RRB- that keep track of recently examined solutions . The second mechanism makes use of one or several memories , which may be referred as long term memories , to direct the search either into the exploration of a promising neighborhood -LRB- intensification -RRB- , or towards previously unexplored regions of the solution space -LRB- diversification -RRB- . It is noteworthy that these memory mechanisms may be viewed as learning capabilities that gradually build up images of good or promising solutions . In the case of the task scheduling problem , the cost of a solution is given by its makespan . The neighborhood N -LRB- s -RRB- of the current solution s is the set of all solutions differing from s by only a single assignment . If ? s 2 N -LRB- s -RRB- , then there is only one task ti 2 T for which As -LRB- ti -RRB- < > A?s -LRB- ti -RRB- . A move is then the single change in the assignment function that transforms a solution s into one of its neighbors . Each move is characterized by a vector -LRB- As -LRB- ti -RRB- ; ti ; pl ; pos -RRB- , associated with taking out task ti 2 T from the task list of processor As -LRB- ti -RRB- and transferring it to that of pl 2 P in position pos . However , the number of neighbor solutions to be examined may be reduced by investigating only a few moves to some positions which most likely will lead to the best neighbor solution -LRB- candidate list -RRB- . The most likely position is obtained through a dynamic task enumeration scheme , which is repeatedly applied each time the makespan of the solution is calculated -LSB- 16 -RSB- . If the best move takes the current solution s to a best neighbor solution s degenerating its cost function , i.e. c -LRB- s -RRB- > = c -LRB- s -RRB- , then the reverse move must be made tabu during a certain number of iterations in order to avoid cycling . In our previous work -LSB- 16 -RSB- , we proposed the tabu search algorithms described in Figures 2 and 3 to solve the scheduling problem defined in Section 2 . A tabu configuration pattern was then defined to be a fully determined set of tabu parameters and implementation strategies . The proposed algorithm makes use of a simple tabu scheme and explores different tabu configuration patterns , each of which involving -LRB- i -RRB- a different type of candidate list , -LRB- ii -RRB- a different type of tabu list , -LRB- iii -RRB- a different value for maxmoves , which determines the maximum number of moves without improvement allowed during the search , -LRB- iv -RRB- a different value for nitertabu , which determines the number of iterations along which a move will be considered as tabu -LRB- i.e , prohibited -RRB- , and -LRB- v -RRB- a different aspiration criterion , i.e. a special condition that , when satisfied by certain tabu moves , allow them to be accepted disabling their tabu status . A series of different tabu configuration patterns were studied side-by-side with a variety of task precedence graphs -LRB- topology , number of tasks , serial fraction , service demand of each task -RRB- and system configurations -LRB- number of processors , architecture heterogeneity measured by the processor power ratio -RRB- . The algorithm showed to be very robust and obtained very good results , systematically improving by approximately 25 % the makespan of the solutions obtained by the best greedy algorithm used as initial solution . 4 Parallel Strategies for the Task Scheduling Problem According to Crainic et al. -LSB- 3 -RSB- , a great number of tabu search procedures may be derived using different strategies when parallel implementations are contemplated . Parallel architectures algorithm tabu-schedule begin Obtain the initial solution s0 Let nitertabu be the number of iterations during which a move is considered tabu Let nmaxmoves be the maximum number of iterations without improvement in the best solution Let tabu be a matrix which keeps track of the tabu status of every move f initialization g s ; s ? s0 Evaluate c -LRB- s0 -RRB- iter 1 nmoves for -LRB- all ti 2 T and all pl 2 P -RRB- do tabu -LRB- ti ; pl -RRB- f perform a new iteration as long as the best solution was improved in the last maxmoves iterations g while -LRB- nmoves < maxmoves -RRB- do begin f search for the best solution in the neighborhood g obtain-best-move -LRB- tk ; pj -RRB- f move to the best neighbor g Move to the neighbor solution s0 by applying move -LRB- tk ; pj -RRB- to the current solution s : set As0 -LRB- tk -RRB- pj and As0 -LRB- ti -RRB- As -LRB- ti -RRB- 8i = 1 ; : : : ; n with i < > k c -LRB- s0 -RRB- c -LRB- s -RRB- + bestmovevalue f update the best solution g if -LRB- c -LRB- s0 -RRB- < c -LRB- s ? -RRB- -RRB- then begin s ? s0 nmoves end f otherwise , update the number of moves without improvement g else nmoves nmoves + 1 s s0 iter iter + 1 end end tabu-schedule Figure 2 : Tabu search algorithm tabu-schedule for the task scheduling problem allow more efficient exploration of the solution space . Generally , this extra efficiency may be achieved by accelerating some phases of the algorithms , or by redesigning them . Several implementations have been proposed in recent literature for the parallelization of tabu search -LSB- 1 , 2 , 4 , 5 , 6 , 7 , 18 -RSB- . As mentioned by Fiechter -LSB- 6 -RSB- , synchronous parallelization schemes generally require extensive communication and therefore are only worth applying to problems in which the computations performed at each iteration are complex and time consuming . This is exactly the case of our scheduling problem , in which the search for the best move at each iteration is a computational intensive task . Due to the presence of the precedence constraints , the calculation of the cost of each solution implies in running an O -LRB- n 2 -RRB- time complexity computational procedure for makespan evaluation , similar to a deterministic event simulation of the execution of the parallel application . Thus , from this point of view , the sequential tabu search algorithm for this problem is suitable for parallelization . Moreover , the size of the problems amenable to be solved in reasonable computational time by sequential tabu search is rather limited and may certainly be increased by the use of a parallel scheme on a faster , more powerful parallel procedure obtain-best-move -LRB- tk ; pj -RRB- begin bestmovevalue 1 f scan all tasks g for -LRB- all ti 2 T -RRB- do for -LRB- all pl 2 P j pl < > As -LRB- ti -RRB- -RRB- do f check whether the move is admissible or not g if -LRB- tabu -LRB- ti ; pl -RRB- < iter -RRB- then begin Obtain the neighbor solution ? s by applying move -LRB- ti ; pl -RRB- to the current solution s : set A?s -LRB- ti -RRB- pl and A?s -LRB- tr -RRB- As -LRB- tr -RRB- 8r = 1 ; : : : ; n with r < > i movevalue c -LRB- ? s -RRB- ? c -LRB- s -RRB- f update the best move g if -LRB- movevalue < bestmovevalue -RRB- then begin bestmovevalue movevalue k i j l end end end end f update the short term memory function g if -LRB- bestmovevalue > = 0 -RRB- then tabu -LRB- tk ; As -LRB- tk -RRB- -RRB- iter + nitertabu end obtain-best-move Figure 3 : Procedure obtain-best-move or distributed computer . The design of parallel implementations for tabu search may use some basic ideas derived from the taxanomy presented in the work of Crainic et al. -LSB- 3 -RSB- , whose main interest is to take into account the differences in control and communication strategies which are so important when designing parallel algorithms . The proposed taxonomy has a twofold basis : first , on how the search space is partitioned ; and second , on the control and communication strategies used by parallel tabu search procedures . The parallelization schemes proposed in this work are synchronized at the end of each iteration of the search . The search for the best neighbor during each iteration is performed in parallel and different sets of neighbor solutions are analysed by each task . This typically characterizes a strict domain decomposition parallelization scheme . Two basic models are used , namely Master-Slave -LRB- MS -RRB- and Single-Program-MultipleData -LRB- SPMD -RRB- , which mainly differ in the way information is exchanged between parallel tasks at the end of each iteration of the tabu search . Four different strategies derived from these two models are described with some detail in what follows . In order to refer to the tasks that compose the parallel program -LRB- described by the task precedence graph -RRB- associated with the task scheduling problem to be solved , we call them in the following by problem-tasks . The tasks which effectively compose the parallel implementation of the tabu search algorithm , and which are distributed to the different processors , are called differently depending on their role in each strategy , namely by slave-task , master-task , parent-task , child-task or simply tasks . 4.1 Master-Slave Model For each current solution s , its neighborhood is partitioned and a set of neighbor solutions to be evaluated must be given to each slave-task . The most balanced way of dividing the neighborhood is to assign an equal number of problem-tasks to each slave-task . The latter will then only consider neighbor solutions obtained through moves of these pre-determined problem-tasks . The search for the best neighbor performed by each slave over a partition of the neighborhood is then called a best neighbor partial search . Moreover , as it will be further explained , there is a choice in some cases of also giving a partition to the master , so that it can also be doing best neighbor partial search , instead of being idle while slave-tasks are executing their search procedures . The number and size of partitions still offer different approaches to this strategy : ffl Single Partitioning -LRB- MS-SP -RRB- : The neighborhood is partitioned only once in approximately equal size partitions , depending on the number of slaves . The partitions of the neighborhood are distributed only once to all the slave-tasks , before the search starts . The master also receives one of the partitions and executes as if it had an embedded slave-task . The master initially distributes the problem and tabu configuration patterns . All tasks -LRB- master and slaves -RRB- use the same method to obtain the same initial solution . The master initializes the current and best solutions as the initial solution , as far as the slaves do not need to keep track of the best solution . During the search , at each time a slave-task finishes its partial search , it sends to the master-task its best local non-tabu move , the cost of the corresponding best local neighbor , and a flag indicating if it eventually did not find any non-tabu move . The master compares its own partial best move with the values received from the slaves and proceeds as in the sequential version : it selects the move corresponding to the best neighbor solution , updates the best solution found so far and its cost , and verifies if the reverse move should be made tabu . If the termination condition is not verified , then it broadcasts the selected move to the slave-tasks , the tabu status of the reverse move and the cost of the eventually new best solution . The master and the slaves update their tabu lists , and restart the search with the same partitions . However , if any non-tabu move has not been found , the master sends to the slave-tasks a tabu list reinitialization message and another search within this same iteration will take place . In case of termination , the master sends to its slaves a final message , so that they can exit . Although the search in each task -LRB- master or slave -RRB- is done partially , their tabu lists are complete because they receive at each iteration the best move selected by the master with its corresponding reverse move tabu status . ffl Multiple Partitioning -LRB- MS-MP -RRB- : In order to meet load balancing requirements , when there are noticeable computational power differences between processors due to machine heterogeneity , load and contention discrepancies , the partition distribution may be done in a work-demand basis . Initially , the neighborhood is partitioned into equal parts as in the former case , but the number of partitions is sufficiently larger than the number of slave-tasks . The initial partitions are distributed -LRB- one per slave -RRB- by the master-task and kept by its slaves . Other initializations are done in the same way as before . During the search , each time a slave-task finishes its work , it sends to the master its partial search results . When the master receives results from a slave and there are still more partitions to be distributed , it sends to this slave a new partition . When all partitions have been searched , the master already has the value of the best neighbor . It then proceeds as in the former case . In this way , more work is given to slave-tasks which are less loaded . On the other hand , it increases the communication between master and slaves , which may in turn reduce the gain obtained from the improved load balancing . It should be noticed that the size of the partitions is fixed and established before the search starts . In this case , as the master must always be available to send new partitions to the slave-tasks , it is not interesting to have it also doing best neighbor partial search as in the MS-SP strategy . 4.2 Single-Program-Multiple-Data Model In this model , tasks work also in rigid synchronization , but there is no master-slave relationship as in the former strategies . All tasks execute the same code -LRB- SPMD model -RRB- following a tokenring communication structure , and the differentiation in the code stands for the task which spawns the others and makes the initialization of the token ring , which we call the parent-task . Tasks communicate pairwise , not strictly between parent and child . The principle here lies on the communication scheme . Tasks are organized in a logical ring and communicate according to this logical circular order established by the parent-task during the spawning procedure . This strategy can be considered according to two different approaches , namely single token and multiple token , due to their resemblence to the equally named access level of local network protocols : ffl Single Token -LRB- SPMD-ST -RRB- : Initialization is done similarly to the former strategies . The parent-task reads problem and tabu configuration pattern files , spawns the other tasks , and broadcasts the necessary information . After the partitions have been sent , each task starts its best neighbor partial search . This strategy is very similar to the master-slave with single partitioning , because also here , all tasks work in parallel in the best neighbor partial search and receive identical size partitions . The difference relies mainly in the comparison step , which in this case is performed in a decentralized way , as explained in what follows . To complete each iteration , the parent-task takes the initiative of sending to its successor the best move value it obtained . This initiative is responsible for starting the communication between tasks along the ring . All tasks wait to receive the best partial move from its predecessor in the logical ring . As each task receives the best move computed by its predecessor , it compares this move with its own result and passes on the best one to its successor . So , when the parent-task finally receives its predecessor message , closing the logical ring cycle , it knows that this is the best move , because it already passed by all other tasks around the ring . Then , it sends forward this best move to its successor , and subsequently do the other tasks , completing a second cycle of message passing around the logical ring . At the end of these two cycles , all tasks have the best move for that iteration . After receiving the global best move , each task proceeds independently updating the best solution found and its cost , the tabu list , the current solution and its cost , the number of iterations without improvement and the iteration counter . Each one then verifies the termination condition . If the search is to proceed , the next iteration is initiated immediately after the updating . ffl Multiple Token -LRB- SPMD-MT -RRB- : The initialization procedure is the same as for the singletoken approach . The difference occurs at the end of the best neighbor partial search . Instead of waiting for the parent-task to initiate the best partial move message passing along the logical ring , each task sends its own result to its successor . When a task receives a result from its predecessor , it compares this information with its own result and sends the best one to its successor . Suppose there is a total of M tasks in the ring . Then , after M ? 1 messages have been received by each task , each of them necessarily has the best global move for this iteration . The updating phase follows , exactly in the same way as for the single token scheme . 5 Computational Experiments In this section , we describe the framework for the computational experiments presented further . The hardware platform used for the implementation and performance evaluation of the parallel implementations of tabu search for the task scheduling problem is the IBM 9076 Scalable POWERparallel 1 -LRB- IBM SP1 -RRB- machine working under PVM . The PVM package was used as the communication platform , due to its portability and flexibility . In terms of performance , as a tradeoff to portability , generally PVM does not profit from the hardware communication facilities available on specific parallel machines . However , there is a trend towards the design of parallel programs using more general platforms , in order to follow the speed of change in the computer system market . In this sense , PVM has been conforming itself as one possible standard in parallel programming development environments . 5.1 Configuration of the Test Problems The characteristics which fully describe the scheduling problem are the same as those in Porto and Ribeiro -LSB- 16 -RSB- . An instance of our scheduling problem is characterized by the workload model and the system model . A deterministic model is used , in which the precedence relations between the tasks and the execution time needed by each task are fixed and known beforehand -LRB- i.e. , before an assignment of tasks to processors is devised -RRB- . Although deterministic models are unrealistic , since they ignore e.g. deviations in task execution times due to interrupts and contention for shared memory , they make possible the static assignment of tasks to processors -LSB- 17 -RSB- . There is only one heterogeneous or serial processor , which has the highest processing capacity . The remaining m ? 1 processors are called homogeneous or parallel processors . Any processor is able to execute any task , i.e. , they all have the same instruction set . The processor power ratio defined in -LSB- 13 -RSB- measures the ratio between the execution time of any instruction in the homogeneous processor and its execution time in the fastest processor . Then , the heterogeneity of the architecture is measured by the processor power ratio . We consider one unique topology for the precedence graph -LSB- 19 -RSB- , typical of the Mean Value Analysis -LRB- MVA -RRB- solution package for product form queueing networks containing two classes of N customers each -LSB- 12 -RSB- . Figure 4 depicts an example of a task graph associated with the MVA algorithm for N = 4 -LRB- n = 25 tasks -RRB- . This dynamic programming scheme represents several computations qualified as wave front , because the topology is characterized by a regular mesh divided in two distinct phases . The first phase of the computations shows a slowly increasing parallelism , which slowly decreases during the second one . The horizontal central axis is formed by the nodes in the middle horizontal axis of the graph , with the largest number of tasks nh = N + 1 . For this same topology pattern , we define different precedence graphs -LRB- i.e. , different applications -RRB- by varying the size of the graph , given by the number of tasks n = n 2 h . We have taken nine different graph sizes with the number n of tasks ranging from 16 to 400 , corresponding to taking nh ranging from 4 to 20 , by steps of 2 . The service demand of the tasks at the border of the graph is equal to one , while that of the inner tasks equals two -LSB- 20 -RSB- . The processor power ratio was fixed at 5 , while the number m of processors was made equal to one half of the number of tasks in the horizontal axis , i.e. m = nh = 2 . m ? ? @ @m ? ? @ @ m ? ? @ @ increasing parallelism m ? ? @ @ m ? ? @ @ m ? ? @ @m ? ? @ @ m ? ? @ @ m ? ? @ @ m ? ? @ @m ? ? @ @ m ? ? @ @ m @ @ m ? ? @ @ m ? ? m ? ? @ @ m @ @ m ? ? @ @ m ? ? m ? ? @ @ m @ @ m ? ? m @ @ m ? ? decreasing parallelism m Figure 4 : Task graph for an application of the MVA algorithm with two classes of N = 4 customers -LRB- n = 25 tasks -RRB- 5.2 Tabu Configuration Pattern The tabu-schedule algorithm for the task scheduling problem strongly depends on two parameters , namely the tabu tenure nitertabu and the maximumnumber maxmoves of iterations without improvement . Its behavior also depends on the strategy implemented for the type and restrictiveness of the tabu list , as well as on the candidate list and aspiration criteria strategies . In Porto and Ribeiro -LSB- 16 -RSB- , several experiments were made in order to obtain the best tabu configuration pattern , which would provide the best performance for the tabu-schedule algorithm . This study was performed based on an application with the MVA topology , with the number of tasks in the horizontal axis ranging from 6 to 14 -LRB- accordingly , the number of tasks ranges from 36 to 196 -RRB- . We have retained the following tabu configuration pattern as the outcome of these experiments : -LRB- i -RRB- candidate list built through a dynamic enumeration technique determining only one single position for the moving task in the task list of each target processor , -LRB- ii -RRB- tabu list organized as matrix , in which each element -LSB- i ; j -RSB- holds the last iteration value until which the move of task ti 2 T to processor pj 2 P is prohibited -LRB- i.e. , tabu -RRB- , -LRB- iii -RRB- maxmoves = 100 , -LRB- iv -RRB- nitertabu = 20 , and -LRB- v -RRB- an aspiration criterion establishing that a certain tabu move drops its tabu classification if it takes the current solution s to a neighbor solution which improves the best solution s ? found so far . 5.3 Parallel Strategies We define the parallelization level -LRB- q -RRB- of a given implementation of some parallel algorithm as the number of processors which work together throughout its execution . In the case of the master-slave strategies q = 1 + nslaves , where nslaves stands for the number of slave-tasks . For the SPMD strategies q = 1 + nchildren , where nchildren stands for the number of childtasks which compose the logical ring together with the single parent-task . The computational experiments have been performed for different parallelization levels -LRB- q -RRB- , which represent the number of processors executing the parallel algorithm . The MS-MP strategy is also characterized by the partition size . The smaller the partition size , the smaller the granularity of the work dispatched to the slaves by the master . Load balance may be done more accurately , although more communication will take place during each iteration until the whole neighborhood is searched . Consequently , there will be a threshold for the partition size , beyond which good performance will not be attained . This threshold may depend not only on the problem size and parallelization level , but also on external factors such as the current system load . We have partitioned the neighborhood in 2 ? nslaves partitions in the case of MS-MP , corresponding to an average granularity of two partitions per slave . 5.4 Numerical Results The performance of some parallel strategy may be evaluated by its speedup S and its efficiency ? defined as : S = oseq opar and ? = S q ; where oseq and opar are , respectively , the sequential and parallel elapsed times observed for the sequential algorithm and this parallel strategy for the tabu search algorithm . The computational results obtained on the IBM SP1 machine are presented in Figures 5 to 7 . Linear speedup is attained when the efficiency is equal to one , i.e. the speedup is equal to the parallelization level q . The closer the efficiency is to one , the more the parallelization scheme benefits from the system parallelism . Differences in code and data distribution for sequential and parallel versions of the algorithm may generate distinct memory access latencies . Moreover , other applications in the system impact differently the performance of the sequential and parallel tabu algorithms . The sequential tabu version is more affected by other applications , as far as it is executed on a single processor . We notice that the sequential programs , master-tasks and parent-tasks are executed on the front-ends , which were the only nodes externally accessible -LRB- and consequently more loaded -RRB- in the system during most of the time , while slave or child tasks are allocated to the less loaded nodes . Also , as the parallel algorithm presents communication-computation overlap , the overhead due to the communication introduced by the parallelization is minimized as a factor of performance degradation . As a consequence , efficiency values greater than one were obtained in some experiments with higher problem size values . On the other hand , we can observe very low efficiency values for small size problems . In these cases , the neighborhood partitions distributed among parallel tasks are not sufficiently large to overcome the embedded overhead due to synchronization between cooperative tasks in the parallel algorithm . This effect is even greater for large values of q , which also contribute to decrease the sizes of the neighborhood partitions . Because the efficiencies have been calculated using the elapsed times , the discrepancies in Figures 5 to 7 seem to be due to the fact that the different cases were run throughout a long period of time , during which the machine was subject to different workloads , consequently producing a different impact on the elapsed time of each test problem and strategy . Figure 5 illustrates the behavior of the efficiency according to the problem size for different parallelization levels , in the case of MS-SP and MS-MP strategies . Both strategies present very similar results . However , the load balancing aspect of the MS-MP strategy is significative if one considers that , in this scheme , the master-task does not execute any partial best neighbor search . The solution neighborhood to be searched is divided solely among slave-tasks . The master rests as a partition distributer and is in charge of the comparison and election of the best move at each iteration . Thus , the importance of load balancing is so effective that it compensates by far the decrease in the computational power dedicated to the best neighbor search . In this case , the comparisons between successive partial best moves is immediate , because the master is completely dedicated to the reception of the results sent by the slaves . 2 4 6 8 10 12 14 16 18 20 nh 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 ? MS-SP ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... . ... ... ... ... ... ... ... ... ... ... . q = 6 q = 12 q = 16 2 4 6 8 10 12 14 16 18 20 nh 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 ? MS-MP ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... . q = 6 q = 12 q = 16 Figure 5 : Efficiency -LRB- ? -RRB- vs. problem size -LRB- nh -RRB- : MS-SP and MS-MP strategies We present in Figure 6 the behavior of the efficiency according to the parallelization level for different problem sizes , in the case of the MS-SP strategy . We notice from these figures that the efficiency systematically -LRB- i -RRB- increases with the problem size for the same parallelization level and -LRB- ii -RRB- decreases with the parallelization level for the same problem size , due to the decrease in the communication/computation ratio . Figure 7 shows performance results for the SPMD-ST strategy . The comparisons cycle is dependent on the parent-task initiative and on the natural order of the ring . Each childtask only sends the message with its own best partial move when it has already received a message from its predecessor in the ring . However , due to the small size of the messages and to the fast communication media , the two cycles of message exchange do not produce high overhead . The performance gain due to the overlap between communication and computation is sufficient to compensate the rigid synchronism established by this strategy . Similar results were obtained with the SPMD-MT strategy The greater number of messages in the case of SPMD-MT compensates higher communication synchronization in SPMD-ST . 2 4 6 8 10 12 14 16 q 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ? ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... . . ... ... ... ... ... . ... ... ... ... ... ... ... ... ... ... ... ... ... . . ... ... ... ... ... ... ... . nh = 10 nh = 12 nh = 14 nh = 16 Figure 6 : Efficiency -LRB- ? -RRB- vs. parallelization level -LRB- q -RRB- : MS-SP strategy The interesting aspects of the logical ring communication scheme , following either a single token or a multiple token approach , are the decentralization of the search and the parallelization of the procedure for the comparison of the best partial moves . Although the logical ring communication scheme could apparently be more promising in terms of the overall performance , specially in the multiple token case , where many pairs of tasks may communicate simultaneously , it should be noticed that this advantage is totally dependent on the architecture of the parallel machine . The standard PVM framework used for implementing these strategies offers a broadcast primitive which actually acts physically as a sequence of pointto-point transmissions in terms of transmission delay , although the sender does not build the same message many times in the output buffer . It does not profit from the architectural characteristics to perform message diffusion . Thus , communication on the IBM SP1 under PVM is done as if processors were workstations linked by a local Ethernet network . The access method to the communication media in this topology assures that only one communication will be in execution at each time . In this case , it may be possible that no profit results from the use of the logical ring communication scheme . 6 Conclusions We have presented four different parallelization strategies for a task scheduling algorithm on heterogeneous processors under precedence constraints , based on a tabu search approach . All strategies are synchronous and their differences rely exclusively on distinct information communication patterns between parallel tasks during execution . The parallel programs were implemented on a IBM SP1 machine under PVM for varying problem sizes and parallelization levels . The computational results confirm the great adaptability of this kind of algorithm to parallelization , showing that communication is not a burden to the achievement of almost ideal efficiency in the majority of the studied cases . Moreover , good performance results in parallelizing other tabu search algorithms have also been reported in recent literature , suggesting that this may be an interesting research path to overcome the inherent intensive computational demand of this kind of algorithm , giving way to the exploration of other aspects such as more intricate tabu search features and asynchronous parallelization schemes . 2 4 6 8 10 12 14 16 18 nh 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 ? ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... . q = 6 q = 12 q = 16 Figure 7 : Efficiency -LRB- ? -RRB- vs. problem size -LRB- nh -RRB- : SPMD-ST strategy References -LSB- 1 -RSB- J. Chakrapani and J. Skorin-Kapov , \ Massively Parallel Tabu Search for the Quadratic Assignment Problem '' , Technical Report HAR-91-06 , Harriman School , Stony Brook , 1992 . -LSB- 2 -RSB- J. Chakrapani and J. Skorin-Kapov , \ Mapping Tasks to Processors to Minimize Communication Time in a Multiprocessor System '' , Working paper , 1993 -LSB- 3 -RSB- T.G. Crainic , M. Toulouse and M. Gendreau , \ Towards a Taxonomy of Parallel Tabu Search Algorithms '' , Research Report CRT-933 , Centre de Recherche sur les Transports , Universit?e de Montr?eal , 1993 . -LSB- 4 -RSB- T.G. Crainic , M. Toulouse and M. Gendreau , \ A Study of Synchonous Parallelization Strategies for Tabu Search '' , Research Report CRT-934 , Centre de Recherche sur les Transports , Universit?e de Montr?eal , 1993 . -LSB- 5 -RSB- T.G. Crainic , M. Toulouse and M. Gendreau , \ An Appraisal of Asynchronous Parallelization Approaches for Tabu Search Algorithm '' , Research Report CRT-935 , Centre de Recherche sur les Transports , Universit?e de Montr?eal , 1993 . -LSB- 6 -RSB- C.-N . Fiechter \ A Parallel Tabu Search Algorithm for Large Traveling Salesman Problems '' , Discrete Applied Mathematics 51 -LRB- 1994 -RRB- , 243 -LCB- 267 . -LSB- 7 -RSB- B. Garcia and M. Toulouse , \ A Parallel Tabu Search for the Vehicle Routing Problem with Time Windows '' , to appear in Computers and Operational Research , 1994 . -LSB- 8 -RSB- F. Glover , \ Tabu Search -LCB- Part I '' , ORSA Journal on Computing 1 -LRB- 1989 -RRB- , 190 -LCB- 206 . -LSB- 9 -RSB- F. Glover , \ Tabu Search -LCB- Part II '' , ORSA Journal on Computing 2 -LRB- 1990 -RRB- , 4 -LCB- 32 . -LSB- 10 -RSB- F. Glover and Manuel Laguna , \ Tabu Search '' , Chapter 3 in Modern Heuristic Techniques for Combinatorial Problems -LRB- C.R. Reeves , ed . -RRB- , 70 -LCB- 150 , Blackwell Scientific Publications , Oxford , 1992 . -LSB- 11 -RSB- F. Glover , E. Taillard , and D. de Werra , \ A User 's Guide to Tabu Search '' , Annals of Operations Research 41 -LRB- 1993 -RRB- , 3 -LCB- 28 . -LSB- 12 -RSB- M. Reiser and S.S. Lavenberg , \ Mean Value Analysis of Closed Multichain Queueing Networks '' , Journal of the Association for Computing Machinery 27 -LRB- 1980 -RRB- , 313 -LCB- 322 . -LSB- 13 -RSB- D.A. Menasc ? e and V. Almeida , \ Cost-Performance Analysis of Heterogeneity in Supercomputer Architectures '' , Proceedings of the Supercomputing '90 Conference , New York , 1990 . -LSB- 14 -RSB- D.A. Menasc ? e and S.C.S. Porto , \ Processor Assignment in Heterogeneous Parallel Architectures '' , Proceedings of the IEEE International Parallel Processing Symposium , 186 -LCB- 191 , Beverly Hills , 1992 . -LSB- 15 -RSB- S.C.S. Porto , Algoritmos Heur ?? sticos para o Escalonamento de Tarefas em Multiprocessadores Com Arquitetura Heterog ^ enea : Constru?c ~ ao Sistem?atica e Avalia?c ~ ao de Desempenho M.Sc dissertation , Catholic University of Rio de Janeiro , Department of Computer Science , Rio de Janeiro , 1991 . -LSB- 16 -RSB- S.C.S. Porto and C.C. Ribeiro , \ A Tabu Search Approach to Task Scheduling on Heterogeneous Processors under Precedence Constraints '' , International Journal of High Speed Computing 7 -LRB- 1995 -RRB- , 45 -LCB- 71 . -LSB- 17 -RSB- M.J. Quinn , Designing Efficient Algorithms for Parallel Processors , McGraw-Hill , New York , 1987 . -LSB- 18 -RSB- E. Taillard , \ Robust Tabu Search for the Quadratic Assignment Problem '' , Parallel Computing 7 -LRB- 1991 -RRB- , 443 -LCB- 455 . -LSB- 19 -RSB- J. Zahorjan and C. McCann , \ Processor Scheduling in Shared Memory Multiprocessors '' , Technical Report 89-09-17 , Department of Computer Science and Engineering , University of Washington , 1989 . -LSB- 20 -RSB- J. Zahorjan , personal communication , 1992 .

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

parallel algorithms
spmd
tabu search
task scheduling