Akademie v < = ed < = Cesk?e republiky ? ustav teorie informace a automatizace Academy of Sciences of the Czech Republic institute of information theory and automation RESEARCH REPORT J.J. Hench The Pseudo-Givens Rotation No. 1800 February 1994 ? UTIA AV < = CR P. O. Box 18 , 182 08 Prague , Czech Republic Telex : 122018 atom c , Fax : -LRB- +42 -RRB- -LRB- 2 -RRB- 6641 4903 e-mail : utia@utia.cas.cz This report constitutes an unrefereed manuscript which is intended to be submitted for publication . Any opinions and conclusions expressed in this report are those of the author -LRB- s -RRB- and do not necessarily represent the views of the Institute . The Pseudo-Givens Rotation ? J.J. Hench Institute of Information and Control Theory Academy of Sciences of the Czech Republic Pod vod?arenskou v < = e < = z ?? 4 128 08 Praha 8 tel : -LRB- 42 -RRB- -LRB- 2 -RRB- 6605-2342 e-mail : hench@utia.cas.cz Abstract A pseudo-Givens rotation is a two by two real matrix that has the following properties . First , it zeros an element in an arbitrary column vector from the left , while its inverse zeros an element in an arbitrary row vector from the right . Second , the remaining degrees of freedom are subject to a minimization that is invariant to inversion and proper orthogonal transformation . One of its interesting properties is that the pseudo-Givens rotation is the Givens rotation in the event that the row vector with an element to be zeroed is the transpose of the column vector with an element to be zeroed . Another is that the pseudo-Givens rotation belongs to the set of matrices with determinant equal to one . In this respect , it is similar to the Givens rotation and the Hyperbolic rotation . 1 Introduction The Givens transformation is one of the most basic tools in numerical linear algebra . In fact , many standard algorithms which factor or decompose a ? This research was supported by the Academy of Sciences of the Czech Republic . matrix to canonical form that includes an orthogonal matrix may be based on the Givens rotation -LSB- 1 -RSB- . For example , the Givens transformation is a natural tool for the reduction of symmetric matrices to banded form , since there exists a symmetric banded matrix that is orthogonally similar to a full symmetric matrix . In the case where a matrix is nonsymmetric , the orthogonal transformations can zero either its upper or lower bands , but not both simultaneously . Simultaneous reduction of a non-symmetric matrix requires a 2 ? 2 transformation that satisfies constraints in its row and column space . This paper proposes the pseudo-Givens rotation for this purpose . While the pseudo-Givens rotation is not generally orthogonal , some similarities to the Givens transformation remain . Among these is that its determinant is equal to one . 2 Givens Transformation In this section the Givens rotation is re-examined . The Givens rotation is a 2 ? 2 orthogonal matrix , which may be parameterized by the parameter ? -LSB- 1 -RSB- Q = 2 4 cos -LRB- ? -RRB- sin -LRB- ? -RRB- ? sin -LRB- ? -RRB- cos -LRB- ? -RRB- 3 5 : -LRB- 1 -RRB- This matrix can be viewed the parameterization of a matrix with a positive determinant and one orthogonality constraint and two normalization constraints ; namely , that the two columns -LRB- rows -RRB- have two-norm equal to one and that the second column -LRB- row -RRB- is perpendicular to the first . Clearly , then , the only degree of freedom available that remains is the orientation of the first column -LRB- row -RRB- vector . If we further demand that the following equation is satisfied , Q ? u = ffe1 where u = 2 4 cos -LRB- ? -RRB- sin -LRB- ? -RRB- 3 5 ; e1 = 2 4 1 3 5 ; -LRB- 2 -RRB- for some 2 R with ? = if = and where det -LRB- Q -RRB- = 1 , then these requirements quite naturally lead to the definition of the Givens rotation in -LRB- 1 -RRB- . The aim of this section is to show that it is possible to derive the Givens rotation in a different way . Instead of imposing two normalization con - straints and an orthogonality constraint , the Givens rotation with be the minimum of a cost function over the domain of real two by two matrices . The criteria for acceptability for the cost functions considered by this paper is that they are invariant to proper orthogonal transformation and inversion . The minimization , therefore , will not favor any particular orientation of the column space or the row space . This is important since the orientation of the Givens rotations in most algorithms is determined by the need to zero an element in an arbitrary vector as in -LRB- 2 -RRB- . In the remainder of this section , a cost function is proposed which meets the criteria above . The minimization is designed to minimize the -LRB- two -LCB- norm -RRB- condition number of a matrix . With no external constraints , this minimum will be the Givens rotation . We proceed by proposing these conditions in the following lemma . Lemma 1 Let W ; ? 2 R2 ? 2 , such that W = argmin ae 1 4 trace h ? T ? + ?? T ?? 1ioe -LRB- 3 -RRB- subject to the constraint that Wu = v as in -LRB- 2 -RRB- and det -LRB- W -RRB- > 0 . There exists a Givens rotation , Q such that Q = W . Proof : Let the singular value decomposition of ? = U?V T . Then , trace h ? T ? + ?? T ?? 1i = trace h V ? UTU?V T + U ?? 1V TV ?? 1UTi = oe21 + oe22 + 1oe 21 + 1oe 22 : -LRB- 4 -RRB- where oe1 and oe2 are the singular values of ? . The global minimum of trace h ? T ? + ?? T ?? 1i is attained when oe1 = oe2 = 1 . Since this minimum is attainable with the constraint Wu = v , then W = UV T is an orthogonal matrix . Since W is an orthogonal matrix and since det -LRB- W -RRB- > 0 , W is a Givens rotation . Remark 1 It is noteworthy that the cost function 1 4 trace h ? T ? + ?? T ?? 1i is invariant to proper orthogonal transformations and inversions , since it is equal to the sum of the squares of singular values and their inverses , which is evident in -LRB- 4 -RRB- . 3 Pseudo-Givens Transformation The minimization given in the previous section is noteworthy not only in that it is orthogonally invariant , but also that it does not impose an orthogonality condition upon the columns -LRB- rows -RRB- of the eventual transformation . The motivation for this is as follows . Suppose we are given the constraints on the matrix W such that for an arbitrary vector u Wu = ffe1 uT W ? 1 = ffeT1 -LRB- 5 -RRB- with e1 and defined as in -LRB- 2 -RRB- . What seems like two constraints in -LRB- 5 -RRB- is actually only one , and the optimal W with respect to the cost function in -LRB- 4 -RRB- is a Givens rotation . This type of constraint is found frequently in the reduction of symmetric matrices to similar symmetric banded matrices . Suppose , however , we are given the constraints on the matrix W such that for an arbitrary vectors u and v Wu = ffe1 vTW ? 1 = eT1 : -LRB- 6 -RRB- For arbitrary vectors u and v , it is impossible to insist that the columns -LRB- rows -RRB- of X are orthogonal . However , it is possible to apply the same minimization procedures from the previous section to compute an W subject to the conditions in -LRB- 6 -RRB- . First , we elaborate under what conditions such a transformation may be found . Lemma 2 Let M ; N 2 R2 ? 2 and kvk < > and kuk < > such that Mu = ffe1 vT N = eT1 -LRB- 7 -RRB- with < > and < > 0 . There exist an M and N such that M = N ? 1 if and only if vTu < > 0 . Proof : -LRB- ! -RRB- Suppose M = N ? 1 that satisfies -LRB- 7 -RRB- , Then vTu = eT1 N ? 1M ? 1e1 = eT1 MM ? 1e1 = eT1 e1 = < > : -LRB- -RRB- Since kvk < > and kuk < > 0 , it is possible to find a Givens rotation Q such that Qu = e1 with < > 0 . Let vTQT = w = -LSB- w1 ; w2 -RSB- . Since vTu = vT QTQu = vTQTe1 = w1 ; the condition vTu < > implies w1 < > 0 . Now , let T = 2 4 1 w2w1 1 3 5 -LRB- 8 -RRB- , M = TQ , and N = T ? 1QT Mu = TQu = T e1 = T e1 = ffe1 where = . Similarly , vT N = vTQTT ? 1 = wTT ? 1 = -LSB- w1 ; -LRB- w1w2 = w1 ? w2 -RRB- -RSB- = eT1 ; where = w1 , which completes the proof . In the previous lemma , the matrix M which satisfies -LRB- 6 -RRB- was produced by operating on u first with a Givens rotation to satisfy the first equation and then by finding an upper triangular matrix T to satisfy the second equation . This procedure is used again to find the pseudo-Givens rotation W . Since the minimization procedures are invariant with respect to Givens rotations -LRB- v. Remark 1 -RRB- , it suffices merely to find the constrained minimum of the cost function whose domain is the set of upper triangular matrices . This minimum is elaborated in the following lemma . Lemma 3 Let W ; ? 2 R2 ? 2 , such that W = argmin ae 1 4 trace h ? T ? + ?? T ?? 1ioe -LRB- 9 -RRB- subject to the constraints Wu = ffe1 vTW ? 1 = eT1 with kvk < > 0 , kuk < > 0 , and vT u < > 0 . Also , let Q be a Givens rotation such that Qu = e1 with < > and vTQT = w . Then there exists an upper triangular matrix T such that W = T ? 1Q , with T = 2 4 < = ? w2 < = w1 1 < = 3 5 -LRB- 10 -RRB- where < = = 4 r 1 + w22w21 . Proof : Since kvk < > 0 , kuk < > 0 , and vTu < > 0 , there exists a W that minimizes -LRB- 9 -RRB- . Also , since the minimization is invariant to Givens rotations , it suffices only to perform the minimization on T , which is given by direct substitution . Remark 2 It is particularly interesting to note that along with the Givens rotation and the Hyperbolic rotation -LSB- 1 -RSB- , the resulting pseudo-Givens rotation has determinant det -LRB- W -RRB- = 1 . 4 Conclusion The pseudo-Givens rotation is elaborated in this paper . It resembles the Givens rotation with the exception that it satisfies constraints placed on both its row and column spaces . The pseudo-Givens rotation is calculated by computing the argument that minimizes a certain cost function which is invariant to inversion and rotation . The resulting pseudo-Givens rotation has a number of interesting properties . First , it is exactly the Givens rotation for symmetric constraints . Second , it belongs to the set of matrices with determinant equal to one . It promises to be of use as an alternative to the Givens rotation in producing similar banded matrices and banded eigenvalue revealing forms for non-symmetric matrices . Acknowledgment Illuminating discussions with Professor Volker Mehrmann on this subject are gratefully acknowledged . References -LSB- 1 -RSB- Golub , G.H. and C.F. Van Loan , Matrix Computations , Johns Hopkins University Press , Baltimore , 1983 , -LRB- second edition : 1989 -RRB- .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

givens rotations
orthogonal transformations

hyperbolic rotations
non-symmetric matrices
similarity transformations


Chapter 19 A Parallel Hardware Implementation Timo H?am?al?ainen , Jukka Saarinen , Pekka Ojala and Kimmo Kaski Tampere University of Technology Electronics Laboratory P. O. Box 692 , FIN-33101 Tampere , Finland E-mail : timoh@ee.tut.fi, jukkas@ee.tut.fi, ojala@ee.tut.fi, kaski@ee.tut.fi Abstract . This paper presents implementations of genetic algorithms in a tree shape parallel computer architecture . Different levels of parallelism involved in GAs are studied . In addition , basic models for parallel GAs are considered . The tree shape parallel computer system , GAPA -LRB- Genetic Algorithm Parallel Accelerator -RRB- , is described in detail , including architecture and special hardware for GA computations . Mapping studies are given for centralized and distributed GA models . 19.1 Introduction Genetic algorithms are one of the most interesting novel methods for reducing the search effort and time in optimization tasks . The basically simple concept of artificial evolution is very flexible and can be applied to a variety of problems , also for those which are hard to model mathematically . However , applications may represent very different problem domains and therefore the style of the GA may differ significantly from one application to another . In general , GAs are The complete title of this article : Implementing genetic algorithms in a tree shape computer architecture . 259 260 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 strongly application dependent , resulting an abundance of GA variations and extensions -LSB- 450 -RSB- . In addition to the requirements of the problem , the point of view in GAs is how to enhance the quality of the search and how to reduce the search time . These improvements can be gained to a certain extent by adjusting the algorithm and software , but for example further reduction in the search time requires improvements also in the computing hardware . The first experiments and implementations of GAs were performed in sequential computers , where basic genetic operations were repeated numerous times in sequential loops during execution . These systems are still widely used because of their established programming tools , fairly good cost/performance ratio and ease of use . Unfortunately , the search time with these systems may still be insufficient for large problems . A drawback is , that the parallel nature of GAs can not be fully utilized in sequential computers . For these reasons parallel computation is the most promising way for efficient GA implementations . As a consistent consequence , several parallel genetic algorithms and parallel implementations have been introduced along with the appearance of parallel computers . Parallel computers have been intensively studied and designed during the last two decades and now there are several systems available -LSB- 284 -RSB- . These systems vary from large general purpose computers to extremely dedicated parallel hardware for a particular task or algorithm . Since GAs are very dependent on the problem , general purpose systems have been preferred for flexible mapping of the problem . Examples of such systems are Transputers , Hypercube , MasPar and Connection Machine -LSB- 124 , 335 , 1181 -RSB- . Especially transputers have been favoured , because the overall system can be built easily with inexpensive , modular chips . Other general purpose systems typically offer a number of processors and some programmable interconnection network . Dedicated parallel hardware , however , may offer even more performance than general purpose systems . Faster operation can be realized by implementing operations or even whole phases of an algorithm in hardware . On the other hand , dedicated hardware may be more cost-effective , since all features of a large general purpose system are even not needed for a particular task . In this paper , we introduce our GAPA -LRB- Genetic Algorithm Parallel Accelerator -RRB- system , in which the features of flexible operation of general purpose systems and fast processing capability of dedicated systems has been combined . Flexible operation is due to expandable tree shape architecture and programmable processing units , which ensures that several types of GAs can be mapped to the system . Good cost/performance ratio has been achieved by using commercial and common Digital Signal Processors -LRB- DSPs -RRB- and Field Programmable Gate Array -LRB- FPGA -RRB- chips . Before we go on details with our GAPA system , we start by first studying a typical sequential genetic algorithm and what are the most time consuming phases in it . After that we introduce the levels of parallelism in GAs and how to utilize these levels in parallel computation . We also present different models for parallel H?am?al?ainen et al : A Parallel Hardware Implementation . 261 GA and implementations in parallel computers . The implementation in a tree shape computer architecture is given in more detail , whereafter we consider the structure and function of our GAPA system , including special hardware solutions for GA operations . Finally , we show how to map different types of GAs to the GAPA . 19.2 Sequential genetic algorithms The flow of computation in a typical sequential GA is illustrated in fig. 19.1 . In the first phase the problem is encoded to an individual . The actual computation starts in the next phase , in which a number of randomly generated individuals are created to form an initial population . After that , the fitness of each individual is evaluated . If a sufficient solution is found , the GA execution is terminated . Otherwise , some method is used to select parents among individuals for the reproduction phase . In this phase a new generation of the population is created or the old population is updated . After that previous phases are repeated . Usually the loop of fitness evaluation , reproduction and replication have to be performed numerous times before a satisfactory solution has been found . The more time one phase consumes , the larger will be the total search time . On the other hand , the larger the population and the longer the individual strings , the more time each phase takes . Although the time needed for each phase is dependent on the problem as well as computing platform , some general estimations can still be given . The initialization phase is needed only once at the beginning of the execution and thus it has minor contribution to the total search time . Instead , the fitness evaluation may be very complicated , for which reason it might take most of the total time . Reproduction consist mainly of string manipulation , which can be performed in sufficient time span . Still the loop structure itself may consume very much time , because of loop counting and control . In order to reduce the total search time one has to try to improve the search quality and/or speed up execution of single phases . The search quality can be adjusted by parameters like mutation rate and crossover probability . Since that is beyond the scope of this paper , we concentrate on the speed-up of computation . In the next section we show how this can be achieved by means of parallel processing . 19.3 Parallel genetic algorithms The outline of sequential GAs discussed in the previous section illustrates the core sequence of high level operations , which have to be performed one after another . However , we can utilize parallel processing in each of these phases by parallelizing operations in several levels . Actually we can even parallelize the high level structure itself by executing several GAs in parallel . Based on these 262 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 observations , a hierarchy of levels of parallelism can be composed . The levels may be illustratively named as follows : calculation level , phase level , population level and process level parallelism . At the lowest level the basic arithmetic and locigal calculations can be parallelized and executed in parallel . For example , all bits of a long string are processed in parallel , without dividing it into pieces . At the phase level parallelism , a whole phase of an algorithm is parallelized , such that there exist no longer any sequential computation within that phase . The fitness evaluation is a good example of such phase . Because the fitness of an individual is not dependent of the fitness of other individuals , all individuals can be evaluated in a truly parallel fashion . These two lowest levels of parallelism can be utilized in sequential GAs , without changing the basic high level structure of the algorithm . At population level parallelism , the GA may be parallelized by dividing the original population into many subpopulations . These subpopulations are quided to evolve and periodically some of the best individuals are exchanged between subpopulations . Actually this means , that we have many copies of the GA running in parallel instead of the original one . At the process level there may be again many GAs executed in parallel , but now these can be different and independent of each other . The population may be the same for all processes , which means that the solution is searched among many separate processes for the same problem . The two highest levels of parallelism are basis for parallel genetic algorithms -LSB- 124 , 464 , 941 , 945 , 1285 -RSB- . Usually parallel GAs have features from several levels of parallelism , for example population level parallelism may be combined with phase level parallelism . The actually realized parallelism depends naturally on the computing platform and parallel processing features of that platform . It should be noted , that parallel GAs have also been simulated in sequential computers , but as was earlier mentioned this leads to an inefficient implementation . Independent of the implementation style , three main categories for parallel GAs can be found . These can be named as centralized , distributed and network models . The centralized style of parallel GA is illustrated in fig. 19.2 . The idea is to perform some basic calculations or phases of the sequential GA in many slave processors , whereas a master processor maintains the actual population . For that reason this is also called as master-slave or farming model . For example , the master sends one ore more individuals to each slave for fitness evaluation and collects results back to itself . After that the master can perform other phases of the algorithm and again send new individuals for fitness evaluation . Distributed GA is a direct application of the population level parallelism , as illustrated in fig. 19.2 . This is sometimes called as an island model , since the population is divided into many subpopulations , each of them living in its own \ island '' . A host is needed to control the exchange of individuals between subpopulations as well as termination of execution when the solution has been found . H?am?al?ainen et al : A Parallel Hardware Implementation . 263 The network model can be derived from the distributed implementation by increasing the number of subpopulations such that there is only one individual per island . In addition , the individuals are physically connected together as illustrated in fig. 19.3 . Possible arrangement of individuals are two dimensional grid , hypercube , ring and ladder -LSB- 335 , 861 , 945 -RSB- . In this model each individual is living in its own site and occasionally interacts with some of its nearest neigbors . This model is also called diffusion model , because good individuals are effectively diffused around the topology of individuals during execution . All of these parallel GA models can be well implemented in parallel computer systems . The centralized model is especially suited for Single Instruction stream Multiple Data streams -LRB- SIMD -RRB- parallel computer architectures , whereas the network model is most conveniently mapped to a Multiple Instruction streams Multiple Data streams -LRB- MIMD -RRB- computer system -LSB- 284 -RSB- . The distributed model may be implemented in either of these architectures , depending on the resources available in the system . There is also many possible topologies for arrangement of processors in these architectures , resulting many choices for GA computation . Although the optimal mapping depends on the target computer system , some general styles of mapping can be given . The centralized GA is well suited for example to an array of processors , where each processor is connected to a master processor via bus . This is reasonable , because the master can not send or collect individuals simultaneously from slave processors . In the case of distributed GA each subpopulation is best mapped to a single processor and one of the processors may be dedicated for hosting purposes . The physical connection between processors need not to be the same as logical connection between subpopulations . The more important thing is , that an individual can be routed to the desired subpopulation by some means . In the network model GA to each processor may be assigned one individual . Usually this means that the topology of processors dictates also the topology of individuals , because there is quite much communication between processors and communication distances should be minimized . The parallel implementation of GAs raises also problems , which are not present with sequential computers . One problem is , how to allocate computer resources with the requirements of the algorithm . The granularity of the algorithm should be suitable for balanced and efficient operation . Fine granularity means that very simple calculations are performed in each processor and there might be very much communication between processors . On the other hand , coarse granularity will require complicated operations or much temporary memory for processors but less communication . Another problem is how to balance the workload between processors in a parallel computer . If the granularity of the algorithm is very fine , the workload might be well balanced , but still the performance is reduced because of communication overheads . On the other hand , when the granularity is too coarse , some processors may have much idle time . For example , in the centralized GA 264 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 implementation the host may be idle when slaves are computing fitness . The control of the whole computation is also a problem . Synchronous operation of processors and tight global control ensures good management of what happens , but this may cause overheads for some processors . For this reason also asynchronous implementations of parallel GA models have been introduced . An example would be asynchronous master-slave implementation , where slaves are computing fitnesses independent of each other and even while the host is performing other phases of the algorithm . In this case there might be individuals from many generations simultaneously and there no longer exist clear boundaries between old and new populations . This is sometimes called as continuous generation implementation . As can be seen , there is many possibilities for parallel GA implementations . The flexibility of GAs offer choices to utilize different levels of parallelism and GAs can be mapped to several parallel hardware platforms . In the next section we introduce a tree shape computer architecture and how genetic operations can be mapped to this architecture . 19.4 Tree shape architecture The tree shape parallel computer architecture is illustrated in fig . ?? . The top of the tree is formed by a horizontal line of processors , the trunk is composed of interconnecting nodes and the root may be a processor or an interface to another system . In general , this architecture can be referred as a master-slave configuration , where the root acts as a master and top processors as slaves . This observation is confirmed by the way of communication , since there is no direct connection between processors and all the communication should be carried via the root . In its basic form the trunk of the tree is only a routing resource , where the interconnecting nodes behave as switches to establish a connection between processors and the root . However , the role of the trunk -LRB- or communication network -RRB- may be extended , if some advanced operations could be included in interconnecting nodes . By this way , the communication network as a whole becomes an active processing element . As processors on top of the tree are dedicated to local operations , the communication network may be used for global operations such as broadcasting , summing , string manipulation or sorting . These features can be utilized at all levels of parallelism , depending on the granularity of the genetic algorithm . To illustrate the possible roles of elements in the tree , let us consider a mapping of some GA operations to this architecture . The fitness evaluation is most conveniently performed in processors , since the fitness function is always dependent on the given problem and processors can be programmed flexibly . But then there is many possibilities to map reproduction phase to the architecture , de - H?am?al?ainen et al : A Parallel Hardware Implementation . 265 pending on the GA model . Especially , the selection of mate pairs for crossover and mutation can be carried in several ways . In some selection methods the population should be sorted , which may be performed in the communication tree . Processors send individuals for comparison and the master or host receives a list of sorted individuals , as illustrated in fig . ?? . In the familiar roulette-wheel selection method a cumulative sum of fitness values of individuals is required . This can be again computed in the communication network , since each interconnecting node in the tree can be configured as an adder , as depicted in fig . ?? . In both of these cases the master selects the mate pairs . Selection can be performed also in slave processors , if the selection is based on local knowledge of fitness values , as is the case with network model GAs . The crossover and mutation phases consists of quite regular string manipulation tasks and can be performed very efficiently in hardware . Now the communication network can be utilized by performing crossover and mutation in interconnecting nodes , as illustrated in fig . ?? . The operation starts by first sending parents from processors down to the communication network . In some interconnecting node the parents meet for crossover and mutation . The resulting offspring or offsprings are then passed further to the master . Reproduction in the tree offers also an interesting variation to this basic scheme of mating two parents . A multiparent mating results , if all processors send individuals to the tree simultaneously and reproduction is performed in all nodes . Naturally the crossover and mutation can be performed also in the master . Slave processors should be used , if the selection was based on local set of individuals . In addition to these possible mappings , the broadcast feature of the communication network can be utilized in all phases , where the same data should be delivered to all processors . The broadcast mode of the tree is illustrated in fig . ?? . The essential question concerning these mappings is , how efficiently global operations in the tree can be performed . In the next section we describe the structure and function of the GAPA system in detail to see how the performance of execution can be enhanced by special hardware . 19.5 GAPA system The GAPA system is based on the TUTNC general purpose neurocomputer , which was designed at Tampere University of Technology in 1993 -LSB- 503 , 743 -RSB- . The GAPA system utilizes the same features of the tree shape computer architecture , but further development has resulted special properties for GA computation . The overall architecture of the GAPA is illustrated in fig . ?? . The processors are named processing units -LRB- PUs -RRB- , the interconnecting nodes in the trunk are called communication units -LRB- CUs -RRB- and the root is an interface -LRB- IFC -RRB- to a host computer . All PUs or CUs are identical to ensure modular structure and simple implementation . The system can be easily expanded by adding PUs and branches 266 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 to the communication network . Based on these architectural considerations , a small scaled prototype of four PUs and three CUs has been realized as shown in the photograph of fig . ?? . In the prototype the main components of the system are a processor board unit and the host computer , which is an IBM PC compatible personal computer . The actual tree shape architecture of PUs and CUs is built in the processor board . In the following we describe in more detail the structure and function of the system . We start by describing the way of processing in the system . After that we explain the structure of PUs and CUs and especially the dedicated hardware for GA computation . The basic modes of communication in the tree are broadcast , write and read , which are data transfer operations . The host can deliver data to all PUs , write data to a particular PU or read data from one PU at a time . The data processing is included in extended modes of communication , which are summation , comparison and reproduction . These modes of operation , linked to the general way of processing in the GAPA , can be illustrated as in fig . ?? . The processing can be modelled as a folded pipeline , which is composed of the processes performed by the host , the communication network and the PUs . The processing units are actually computers of their own , each equipped with a local memory and a general purpose digital signal processor -LRB- DSP -RRB- , as shown in fig . ?? . The DSP is Texas Instruments ' TMS320C25 signal processor -LSB- 622 -RSB- , which was selected for its low cost , good availability and sufficient performance . Its peak multiply-accumulate performance is 20 MIPS in 16 bit , fixed point arithmetic at 40 MHz operation frequency . The program code for the DSP is loaded in the local memory , which is also used for storage of run-time variables . The memory and DSP are connected to the communication network via interfacing circuitry implemented in a single XILINX XC4005 FPGA chip -LSB- 623 -RSB- . In addition , the FPGA chip may be equipped with a parallel hardware random number generator and special reproduction generator , which can be accessed by the DSP . These special pieces of hardware are purposed to accelerate especially string manipulations in the reproduction phase of a GA , if this phase is executed in the PU . Since random numbers are required frequently during GA execution , it is useful to accelerate their generation by hardware . The random number generator can be running continuously and the DSP can simply read a new random number from the generator at any time it requests . The task of the reproduction generator is to get parents from the DSP , perform crossover and mutation and send offsprings back to the DSP . The structure of one CUs is depicted in fig . ?? . CUs are routing switch elements with reduced set of arithmetic and logical functions . In the basic configuration one CU can perform comparison and summation operations in its ALU block . The random number generator and reproduction generator can be also included to a CU , if reproduction is performed in the tree . Data can be routed from the bottom arm to either of the top arms or shared between the left and right arms . In the opposite direction , data can be routed from either of the top H?am?al?ainen et al : A Parallel Hardware Implementation . 267 arms to the bottom arm . In the case of summation , operands are fetched from the top arms and the result is passed down to the next CU or host . In the sorting operation candidates from the top arms are compared and the winner is let to travel down in the tree . Reproduction can be performed such , that parents from the top arms are mated in the reproduction generator and resulting children are passed forward . CUs are implemented with XC4005 FPGA chips , which allow easy reconfiguration for desired functions . Data is buffered and clocked in each CU , wherefore CUs can also be seen as pipeline stages . All routing or processing functions can be performed in a single system clock cycle . As was mentioned , both PUs and CUs can be equipped with special hardware to accelerate GA computation . Reproduction generator and random number generator implement calculation level parallelism , which means that calculations are performed at very low hardware level . At that level there is no longer abstract symbols or strings of characters , but rather a composition of boolean logic operations for strings of bits . In the following we clarify the structure and function of these special generators at hardware level . Since the reproduction is performed in slightly different way in PU and CU , the structure of the reproduction generator is also different for PUs and CUs . The reproduction generator for the PU is shown in fig . ?? . There is a set of registers , which can be accessed by the DSP and multiplexers for parallel string manipulation . Parents to be mated are written to the parent registers and the offspring can be read from its own register . In addition there is two mask registers , which control the crossover points and mutation of the offspring . Each bit position of the offspring register is connected to a 2-input multiplexer , such that a particular bit can be fetched from either of the corresponding bit positions in parent strings . The multiplexer is controlled by the corresponding bit in the crossover mask register . By this way the crossover can be performed in truly parallel fashion , and what is more , the crossover type and number of crossover points can be determined very flexibly . The mutation is performed by inverting random bits in the offspring string . This can be done on the fly by assigning desired points of mutation to the mutation mask register . On the point of view of the DSP , the whole crossover and mutation phase is only composed of computation of masks and writing and reading parent and offspring registers . The computation of masks can be further accelerated by utilizing the random number generator . The registers may be in some cases designed to be as long as individual strings , meaning that there is no need to divide individuals into pieces for actual string manipulation . The structure of the reproduction generator for CUs is a little different from that designed for PUs , as can be seen in fig . ?? . Now the parent strings are fetched from the top arms along with submasks . The idea is to combine the final crossover mask from the submasks , which have been originally computed and sent by different PUs . The control of multiplexers is otherwise similar to that generator designed for PUs . If there is only one parent and submask present , no 268 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 reproduction is performed and the parent is only passed down with the submask . In the case of reproduction only the offspring is passed down , but not any submasks . The following CUs can thus check that the incoming individual is not a parent but an offspring . The mutation operation can be performed in the host computer or autonomously in the CU , which means that no mutation mask is sent by some PU . Instead , CUs are commanded to produce random numbers and invert occasionally bits of the offspring string . The random number generator can be used as stand alone or in conjunction with the reproduction generator , as we have already mentioned . In our system random numbers are generated in Tausworthe pseudorandom number generator , which is based on feedback shift register , as shown in fig . ?? -LSB- 1133 -RSB- . This kind of feedback shift register produce random numbers , which are uniformly distributed and have minimum pair correlations in a long sequence . If the length of the shift register is P , the recurring linear sequence of numbers is at maximum 2P ? 1 . A maximal length sequence can be found by selecting the feedback point Q and shift register length P properly . The actual operation of the random number generator is the following . As a first step the shift register is initiated by assigning randomly chosen seed number . In the next clock cycle bits are shifted to the right and to the first bit position is inserted a modulo-2 sum -LRB- i.e. XOR -RRB- of the two feedback bit positions Q and P . The contents of the last bit position is lost after each shift operation . By this way there are L shifts and XOR operations required for each new L bit random numbers . Sometimes even this takes too much time and can not be tolerated . Faster operation can be achieved by making the modulo additions concurrent , which yields a new random number in each clock cycle . This idea is implemented in our hardware random number generator , which is illustrated in fig . ?? . In this generator there is many short shift registers and a network of XOR elements such that a single shift operation is equivalent to L shifts in the original generator . The new random number can be read from the first bit positions of the shift registers via an interface bus -LRB- see fig . ?? -RRB- . The bus is also used to feed seed number to the generator during initialization phase . In fig . ?? the random number is selected to be 16 bit wide and values for feedback positions Q = 15 are P = 127 . Although there are many random number generators available , the feedback shift register generator was selected for its easy implementation in hardware . A problem with this generator is the initialization phase . Each of the random number generators in PUs and CUs should produce uncorrelated random number sequences , which means that each PU or CU may use only distinct pieces of the available random number sequence . For this reason the first generator can be initiated by a random seed number , but the others have to be initiated based on this first seed number . If there are e.g. four generators , the available random number sequence is partitioned into four subsequencies . Each generator are then initiated to start with one of the four starting points within the original sequence . Several methods have been suggested to solve this problem to determine seed H?am?al?ainen et al : A Parallel Hardware Implementation . 269 numbers for each generator . A review of different methods can be found in -LSB- 1134 -RSB- . We turn now to study the performance of the GAPA system at hardware level to see how efficiently computations can be performed . The performance of the reproduction generator and random number generator can be given in terms of clock cycles needed to accomplish a given task . For example , the reproduction in the PU can be finished in five system clock signals , which is actually minimum time for four write and one read access to the registers . Clocked at 10 MHz , 2 million offsprings can be produced in a second . The actual rate for offsprings is much lower , since the DSP can not feed the registers so fast . With TMS320C25 at minimum 15 system clock cycles are required for read and write accesses , resulting about 600 000 offsprings per second . The computation of crossover and mutation masks can be performed partly simultaneously with reproduction generator . From the random number generator , a new random number can be fetched in each system clock cycle , resulting 10 million random numbers per second . Again the input-output -LRB- IO -RRB- bandwidth of the DSP lowers the available rate , since 3 system clock cycles are required for a read operation , resulting 3.3 million random numbers per second . In the CU each data transfer or processing operation can be accomplished in one clock cycle , resulting 360 Mbit/s data transfer rate through each CU or 10 MIPS performance for summation , comparison or reproduction operations . Since the communication network is pipelined , there can be many data processing operations simultaneously in different CUs . If there is a network of three CUs , the maximum performance is 30 MIPS . If we consider the reproduction phase in the tree , there can be produced 10 million offsprings per second . This can be reached only , if the parents and submasks are computed beforehand and stored to the local memory . From the local memory of the PU data can be transferred with no wait states to the communication network . If the submasks are computed on the fly , the wait states of the DSP should be taken into account . As can be seen , the communication network and special hardware generators can handle data faster than the DSP . This implies , that more performance is gained easily by updating the DSP to a one , which can really throughput data at 10 MHz clock frequency . In the next section we show how different parallel GAs can be executed in the GAPA system , and review the performance issues on the point of execution . 19.6 Mapping of GAs to GAPA The architecture and special hardware features of the GAPA offer several implementation possibilities . In the following , two different mappings are presented , which are based on centralized and distributed style GA models , respectively . In both of these mappings it is supposed that the total number of individuals is N 270 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 and available PU count is P . For simplicity , individual string length is the same as machine word length , i.e. 16 bits . The selection is based on sorting such that the first mate Si is selected from the list of sorted population , beginning from the best individual . The second mate Sj is simply selected such , that j = rand -LRB- m -RRB- ; m 2 -LSB- 0 ; N -RSB- : -LRB- 19:1 -RRB- Only one offspring is generated from two parents , which means that N pairs of mates have to be chosen . The number of individuals is assumed to be same for both implementations . Algorithm for the centralized and distributed implementation is given in table 19.1 and table 19.2 , respectively . In both of tables , operations are sorted along the mapping for PUs , communication network -LRB- tree -RRB- and the host . Initialization is the first phase in both implementations , and it can be carried in PUs by generating random numbers corresponding first individuals . The next phases , which are fitness evaluation and local sorting of population for mate selection , is again performed in PUs for both implementations . Crossover and mutation operations , however , are performed differently . In the centralized implementation the whole population is sorted in the tree whereafter the host selects mate pairs and requests PUs to send individuals for reproduction , which is performed in the tree . In the centralized implementation the host collects offsprings and broadcasts the new population for the next search turn . In the case of distributed implementation subpopulations are independent and each PU performs selection and reproduction locally . Sorting in the tree is performed only for exchange phase and only for some few individuals to be exchanged . To compare these two implementations , we can see that there is quite much idle time for the tree and host in the distributed implementation . On the other hand , there is quite much communication in the centralized implementation and we have faced the familiar trade-off between balance of workload and communication . In addition , it should be noted that these issues do not imply in which implementation style the solution is found faster , since the main structure of the parallel GA is different . For some problems the distributed model is better than centralized , and vice versa . However , we can compare these two implementations in terms of computer machine operations needed , as depicted in table 19.3 . Required operations are the following : random number generation -LRB- denoted TR -RRB- , fitness evaluation -LRB- TF -RRB- , local sorting in each PU -LRB- TSL -RRB- , global sorting in the tree -LRB- TSG -RRB- , selection in the host -LRB- TSH -RRB- , selection in each PU -LRB- TSP -RRB- , computation of crossover and mutation masks -LRB- TM -RRB- , crossover and mutation in the tree -LRB- TCMT -RRB- as well as in the PU -LRB- TCMP -RRB- and broadcast operation -LRB- TB -RRB- . From table 19.3 it can be examined that the initialization and local sorting of individuals requires same operations and same time in both implementations . In the remaining phases there are differences in operations and how many times these are required . The time for sorting operation in the tree is proportional to the height of the tree as well as number of individuals . Selection in the centralized H?am?al?ainen et al : A Parallel Hardware Implementation . 271 implementation depends on the host computer , and can not be considered equally with the selection operation in PUs in distributed implementation . However , the mask computation is performed in PUs in both implementations , wherefore this takes the same time in both cases . Crossover and mutation times are again proportional to the height of the tree in the centralized implementation . In the case of distributed implementation the time for these phases is only proportional to the number of individuals in each PU . Some observations can be done from this discussion . If the number of individuals is kept constant and the number of PUs is increased , the processing time in each PU decreases . However , the tree grows taller presenting more delay , which is , fortunately , not critical for the whole computation time , since the delay is proportional to the 2-base logarithm of the number of PUs . When the number of PUs exceeds the number of individuals , there is any more benefit from adding more PUs . Actually this is not even desirable with distributed implementation , since one individual is not any subpopulation and there should be several individuals per PU . 19.7 Summary and conclusions The architecture and implementation of a parallel hardware platform for GA computation has been introduced . The GAPA is a multiprocessor system with programmable processing units . The number of PUs can be easily increased , such that the desired degree of parallelism and performance can be reached . In addition , the system is modular , since all PUs and CUs are identical . Special hardware generators can be included to both PUs or CUs to accelerate GA computation at low level . Flexibility of operation is due to FPGAs and DSPs , which can be programmed and controlled individually . The cost of the architecture implementation has been kept fairly low , since commercial , popular components have been used . System concept has been tested with a small scale prototype of four PUs , and the architecture with special hardware has been found to be well matched with requirements of different parallel GA models . Mappings of centralized and distributed models of parallel GAs have been presented . Comparisons between mappings have been given in terms of operations required for computation . In the future the performance will be analyzed in detail in order to develop the system further . 19.8 Bibliography See chapter Bibliography at the end of this proceedings . 272 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Figure 19.1 : Operations in a typical GA. . Figure 19.2 : Centralized and distributed models for parallel GA. . H?am?al?ainen et al : A Parallel Hardware Implementation . 273 Figure 19.3 : Network model GA. . Figure 19.4 : Tree shape multiprocessor architecture . 274 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Figure 19.5 : Sorting operation in the tree . Figure 19.6 : Cumulative summation in the tree . H?am?al?ainen et al : A Parallel Hardware Implementation . 275 Figure 19.7 : Broadcast operation . Figure 19.8 : Reproduction in the tree . 276 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Figure 19.9 : Architecture of GAPA computer system . Figure 19.10 : Photograph of GAPA prototype system . H?am?al?ainen et al : A Parallel Hardware Implementation . 277 Figure 19.11 : Operational view of GAPA system . 278 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Figure 19.12 : Block diagram of the PU . Figure 19.13 : Block diagram of the CU . H?am?al?ainen et al : A Parallel Hardware Implementation . 279 Figure 19.14 : The reproduction generator for PU . Figure 19.15 : The reproduction generator for CU . 280 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Figure 19.16 : Feedback shift register method for pseudorandom number generation .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

parallel computer
parallel ga
parallel implementation
tree shape architecture