FORSCHUNGSZENTRUM J ? ULICH GmbH Zentralinstitut f ? ur Angewandte Mathematik D-52425 J ? ulich , Tel. -LRB- 02461 -RRB- 61-6402 Interner Bericht Conjugate Gradients Parallelized on the Hypercube Achim Basermann KFA-ZAM-IB-9309 September 1993 -LRB- Stand 01.09.93 -RRB- Erscheint im `` International Journal of Modern Physics C '' CONJUGATE GRADIENTS PARALLELIZED ON THE HYPERCUBE ACHIM BASERMANN Central Institute for Applied Mathematics Research Centre J?ulich GmbH , 52425 J?ulich , Germany email : acb@zam036.zam.kfa-juelich.de ABSTRACT For the solution of discretized ordinary or partial differential equations it is necessary to solve systems of equations with coefficient matrices of different sparsity pattern , depending on the discretization method ; using the finite element method -LRB- FE -RRB- results in largely unstructured systems of equations . A frequently used iterative solver for systems of equations is the method of conjugate gradients -LRB- CG -RRB- with different preconditioners . On a multiprocessor system with distributed memory , in particular the data distribution and the communication scheme depending on the used data struture are of greatest importance for the efficient execution of this method . Here , a data distribution and a communication scheme are presented which are based on the analysis of the column indices of the non-zero matrix elements . The performance of the developed parallel CG-method was measured on the distributed-memory-system INTEL iPSC/860 of the Research Centre J?ulich with systems of equations from FE-models . The parallel CG - algorithm has been shown to be well suited for both regular and irregular discretization meshes , i.e. for coefficient matrices of very different sparsity pattern . 1 . Introduction For the solution of discretized ordinary or partial differential equations it is necessary to solve systems of equations with coefficient matrices of different sparsity pattern , depending on the discretization method ; using the finite element method -LRB- FE -RRB- results in largely unstructured systems of equations , using the finite difference method results in coefficient matrices of more regular pattern , e.g. banded or block matrices . A frequently used iterative solver for systems of equations is the method of conjugate gradients -LRB- CG -RRB- with different preconditioners .1 ; 2 In 1990 , Aykanat e.a. presented a modified CG-algorithm3 with better parallelization properties than the original method developed by Hestenes and Stiefel . The main work in each iteration of this method consists in the computation of the matrix-vector product . Thereby , accessing the vector is determined by the sparsity pattern and the storage scheme of the matrix . On a multiprocessor system with distributed memory , in particular the data distribution and the communication scheme depending on the used data struture are of greatest importance for the efficient computation of the matrix-vector-product . Here , a data distribution and a communication scheme are presented which are based on the analysis of the column indices of the non-zero matrix elements . First , the matrix is distributed row-wise on each processor , the vector com - ponents accordingly . By analyzing the column indices , each processor determines which matrix elements result in computations with local data and which ones with non-local data . Communication and local computations are performed overlapped to reduce the contribution of the communication time to the total execution time . The data distribution and the communication scheme merely depend on the sparsity pattern of the matrix , not on the values of the non-zero elements . The schemes are determined once before the execution of the CG-iteration . The performance of the developed parallel CG-method was measured on the distributed-memory-system INTEL iPSC/860 of the Research Centre J?ulich with equation systems from two FE-models . The first FE-model comes from environmental science ; it simulates the behaviour of pollutants in geological systems .4 ; 5 In the second FE-model from structural mechanics , stresses in materials induced by thermal expansion are calculated by applying the FE-program SMART .6 2 . The Method of Conjugate Gradients The method of conjugate gradients1 is an algorithm for solving systems of linear equations Ax = b , particularly for sparse coefficient matrices A . The method converges for matrices , which are symmetric and positive definite . Aykanat e.a. 3 suggested a modified CG-algorithm -LRB- see Algorithm 1 -RRB- which has better parallelization properties than the original method . Algorithm 1 . The modified CG-method Choose an arbitrary x0 2 IRn ; g0 = Ax0 ? b d0 = ? g0 i = ; 1 ; : : : i = gTi gi dTi Adi i = i -LRB- Adi -RRB- T Adi dTi Adi ? 1 gTi +1 gi +1 = igTi gi xi +1 = xi + idi gi +1 = gi + iAdi di +1 = ? gi +1 + idi until kgi +1 k2 < = fflr : In each iteration , the vectors xi , gi , and di are computed . xi approximates the solution vector , gi is the residue ; di determines the direction in which the next approximation of the solution vector is searched for . The main work in each iteration consists in the computation of the matrix-vector-product Adi . Furthermore , two dot products and three vector additions have to be performed . Iteration is continued until the euclidean norm of the residue is less than or equal to fflr . Another stopping criterion which uses the maximum scaled absolute difference of the components of the latest two approximations of the solution vector is determined as follows : max j = 1 ; : : : ; n 2 jxji +1 ? xji j jxji +1 j + jxji j < = ffls : -LRB- 2.1 -RRB- The main difference between the modified and the original CG-method is that in Algorithm 1 the constants i and i and therefore all dot products are computed at the beginning of each iteration . If each iteration is performed in parallel on a distributed-memory-system the local values of the dot products can be included in one message for determining the global values .2 3 . Storage Scheme Storage schemes for large sparse matrices depend on the sparsity pattern of the matrix , the considered algorithm , and the architecture of the used computer system . In literature , many variants of storage schemes can be found in e.g. -LRB- Refs . 7-12 -RRB- . The storage scheme considered here is often used in FE-programs and suitable for regular as well as for irregular discretization meshes . It can be found in similar form in e.g. -LRB- Ref . 7 -RRB- . The principle of the scheme is shown in -LRB- 3.3 -RRB- for matrix -LRB- 3.2 -RRB- . The non-zeros of matrix -LRB- 3.2 -RRB- are stored row-wise in three one-dimensional arrays . aw contains the values of the non-zeros , as the corresponding column indices . In az , the position of the beginning of each row in aw and as is stored . The subdivisions in aw and as have been added to mark the beginning of a new row . The order of the matrix elements per row in aw and as is different from that in matrix -LRB- 3.2 -RRB- since this is ususally the case in FE-programs caused by the assembly of the coefficient matrix from the single elements . A = BBBBBBBBBB @ 1 2 9 9 3 10 10 4 11 14 12 18 11 5 17 14 6 15 12 17 15 7 18 8 1 CCCCCCCCCCA : -LRB- 3.2 -RRB- aw = -LRB- 1 j 9 2 j 10 9 3 j 10 4 18 14 12 11 j 11 5 17 j 15 14 6 j 12 17 7 15 j 18 8 -RRB- ; as = -LRB- 1 j 3 2 j 4 2 3 j 3 4 8 6 7 5 j 4 5 7 j 7 4 6 j 4 5 7 6 j 4 8 -RRB- ; az = -LRB- 1 2 4 7 13 16 19 23 25 -RRB- : -LRB- 3.3 -RRB- 4 . Parallelization 4.1 . Data Distribution For parallelizing Algorithm 1 on a distributed-memory-system , the matrix and vector arrays must be suitably distributed to each processor . For the considered data distribution schemes , the arrays aw and as are distributed row-wise ; the rows of each processor succeed one another . The distribution of the vector arrays corresponds component-wise to the row distribution of the matrix arrays . Criteria for the data distribution can be : each processor gets the same number of rows or so many rows that each processor has nearly the same number of non-zeros . The number of arithmetical operations for the computation of the matrix-vectorproduct is proportional to the number of non-zeros ; the remaining arithmetical operations of one iteration are proportional to the number of rows . The criterion considered here is that each processor has to compute nearly the same number of arithmetical operations . If the discretization mesh is regular , i.e. the sparsity pattern of the coefficient matrix is regular , all three criteria result in nearly the same data distribution . If the mesh is very irregular , the three distributions are considerably different . In the latter case , processor k ; k = ; : : : ; p ? 1 , gets so many rows until criterion ek + ? nk e + ? n > = 1 p ; for ek ; nk AE 10 -LRB- 4.4 -RRB- is satisfied for the first time , i.e. for the least number of rows possible . p is the number of the used processors , ek the number of non-zeros , and nk the number of rows of processor k. e is the total number of non-zeros and n the order of the matrix . The parameter ? considers the number of vector-operations besides the operations of the matrix-vector-product and the ratio of the execution times of multiplication , division etc. operations and the addition operation ; it is therefore dependent on the processor architecture . For the INTEL i860XR , ? is about 13 . The numerator in -LRB- 4.4 -RRB- is proportional to the number of arithmetical operations of one iteration of processor k , the denominator is proportional to the total number of arithmetical operations of one iteration . With these considerations , the contribution of the matrix-vector-product to one iteration can be approximated by aMVP ss e e + ? n = 1 1 + ? = mz ; for e ; n AE 10 : -LRB- 4.5 -RRB- mz = e = n is the mean number of non-zeros per row . The data distribution according to criterion -LRB- 4.4 -RRB- is shown in -LRB- 4.6 -RRB- by distributing matrix -LRB- 3.2 -RRB- to four processors . The other arrays are distributed analogously . Processor 0 : aw0 = -LRB- 1 j 9 2 j 10 9 3 -RRB- ; Processor 1 : aw1 = -LRB- 10 4 18 14 12 11 j 11 5 17 -RRB- ; Processor 2 : aw2 = -LRB- 15 14 6 j 12 17 7 15 -RRB- ; Processor 3 : aw3 = -LRB- 18 8 -RRB- : -LRB- 4.6 -RRB- 4.2 . Communication Scheme On a distributed-memory-system , the computation of the matrix-vector-product requires communication because each processor possesses only a part of the components of the vector . For the efficient computation of the matrix-vector-product , it is necessary to develop a suitable communication scheme . First , the arrays ask are analysed on each processor k to determine which data result in accesses to components of di of other processors . Then , ask and awk are rearranged in such a way that the data which result in accesses to processor h are collected in block h . The data of block h succeed row-wise one another with increasing column index per row . Block k is the first block in ask and awk and contains the data which result in local accesses . The goal of the rearranging is performing computation and communication overlapped . The principle of the rearranging is shown in -LRB- 4.7 -RRB- for the data distribution from -LRB- 4.6 -RRB- . Here , merely array as1 is analysed and rearranged . Computing the operation row times vector of the matrix-vector-product of processor 1 , the index 3 results in an access to component d3i of processor 0 , the index 8 to d8i of processor 3 , and the indices 6 and 7 in accesses to d6i and d7i of processor 2 . The data blocks in -LRB- 4.7 -RRB- are separated by double dashes for elucidation ; the blocks have been numbered below the brackets . After rearranging , the data of block 1 result in local accesses , the data of block in accesses to processor 0 , the data of block 2 in accesses to processor 2 , and the data of block 3 in accesses to processor 3 . Processor 0 : as0 = -LRB- 1 j 3 2 j 4 2 3 -RRB- ; di ; 0 = -LRB- d1i d2i d3i -RRB- Processor 1 : as1 = -LRB- 3 4 8 6 7 5 j 4 5 7 -RRB- ; di ; 1 = -LRB- d4i d5i -RRB- Processor 2 : as2 = -LRB- 7 4 6 j 4 5 7 6 -RRB- ; di ; 2 = -LRB- d6i d7i -RRB- Processor 3 : as3 = -LRB- 4 8 -RRB- ; di ; 3 = -LRB- d8i -RRB- Rearranging : as1 = -LRB- 4 5 j 4 5 | -LCB- z -RCB- 1 k 3 | -LCB- z -RCB- k 6 7 j 7 | -LCB- z -RCB- 2 k 8 | -LCB- z -RCB- 3 -RRB- -LRB- 4.7 -RRB- 3 2 1 4 6 74 5 3 8 4 : Processor k : Index j of a vector component k j Fig. 1 . Communication scheme After having analysed the column index array ask , each processor k knows which components of di must be required of which processors . This information is broadcasted to all processors . Then , each processor can decide which data must be sent to which processors . This communication scheme is determined once before starting the parallel CG-algorithm and applies unchanged to each iteration . The communication scheme for the example discussed before is shown in Fig . 1 . Processor 1 e.g. receives the third component of di from processor 0 , the sixth and seventh component from processor 2 and the eighth component from processor 3 . On the other side , the fourth component of processor 1 is sent to processor 0 , the fourth and fifth to processor 2 and the fourth to processor 3 . In Fig. 2 , the parallel computation of the matrix-vector-product is described . First , on each processor , the data which are necessary for other processors are sent asynchronously . After executing asynchronous receive-routines for receiving non-local data , all local computations are performed , in particular the local part of the matrix-vector-product . Then each processor waits until the data of an arbitrary processor arrive and continues the computation of the matrix-vector-product . Thereafter , each processor waits for the data of other processors until the computation of the matrix-vector-product is complete . Computation and communication are performed overlapped . While required data are on the network , operations with local or already arrived data of other processors are executed . 5 . Results The performance of the developed parallel CG-method was measured on the distributed-memory-system INTEL iPSC/860 of the Research Centre J?ulich . The computer system has 32 processors with 16 Megabyte private memory each ; the interconnection network has hypercube-topology . The maximum transfer rate is 2.8 6 Sending the data which are necessary for other processors , asynchronously Receiving non-local data for the matrix-vector-product , asynchronously Local dot product i > 0 ? i > 0 ? i > 0 ? Local computation of maxima for stopping criteria Computing the matrix-vector-product with local data no no no yes yes yes k = 0 1 p-1 ... k = 0 1 p-1 ... Matrix-vector-product with the data of the processor ? ? ? Data of a processor available ? no no no yes yes yes ? ? ? Computation complete ? no no yes yes yes no Fig. 2 . The parallel matrix-vector-product Megabyte/second per channel in both directions . The tests presented here were performed with one equation system each of the FE-models from environmental science and structural mechanics . In Table 1 , numerical data of the coefficient matrices and for the convergence of the CG-method are indicated . The equation system from environmental science has 49392 unknows , that from structural mechanics 25222 . In the first case , the mean number of non-zeros per row is near the maximum number . This is caused by a regular discretization mesh . For the second case , the mean and the maximum number are considerably different ; the discritization mesh is much more irregular . Below in Table 1 , the number of iterations with and without diagonal scaling , a simple preconditioner ,2 is given . The iteration was stopped when the maximum scaled absolute difference from -LRB- 2.1 -RRB- was less than or equal to 10 ? 5 ; this corresponds to a precision of the solution vector of about five decimals . With diagonal scaling , the number of iterations is considerably smaller in both cases . The contribution of this preconditioner to the total execution time is in both cases below 1 % . For the preconditioned method , the euclidean norm of the residue after 84 and 658 iterations , respectively , is given additionally . Table 1 . Numerical data of the considered equation systems Environmental science Structural mechanics Rows 49392 25222 Density 0.05 % 0.6 % Non-zeros per row , max . 27 485 mz 25.2 152.9 aMVP 66 % 92 % Stopping criterion : max . scal . abs . diff . < = 10 ? 5 Iterates without scaling 390 1444 Iterates with scaling 84 658 kgi +1 k2 4:5 ? 10 ? 4 1:5 ? 10 ? 5 The sparsity patterns of both matrices are shown in Figs. 3 and 4 , respectively . The matrix from environmental science has essentially band structure with a maximum bandwidth of 2375 . The matrix from structural mechanics has a much more irregular structure ; the maximum bandwidth is 3474 . In Fig. 5 , execution times per iteration on 32 processors with and without communication and computation performed overlapped are presented . The overlapped execution reduces the execution times by nearly 20 % . In Fig. 6 , speedups on 4 to 32 processors are shown . The equation system from environmental science together with the program code and the remaining data requires the memory of more than two processors , that from structural mechanics the memory of more than four processors . For up to four and , in the second case , up to eight processors , linear speedup was assumed because nearly linear speedup was observed in tests with smaller equation systems for up to 8 processors . For 16 processors , the speedup is 13.2 in the first case and 15.2 in the second case . This corresponds to efficiencies of 83 % and 95 % . For 32 processors , speedups of 21.6 and 27.2 are achieved . The efficiencies decrease to 68 % and 85 % because the communication overhead increases . 6 . Conclusions The developed sequential and parallel CG-algorithms have been shown to be well suited for the considered FE-models . They are used in both projects . On a distributed-memory-system , the developed data distribution and communication scheme together with the overlapped execution of communication and local compu - Fig. 3 . Sparsity pattern of the matrix from environmental science Fig. 4 . Sparsity pattern of the matrix from structural mechanics Fig. 5 . Execution times per iteration tations result in a flexible algorithm . This algorithm performs well for both regular and irregular discretization meshes , i.e. for coefficient matrices of very different sparsity patterns . In future work other data distributions and other storage schemes for sparse coefficient matrices and their influence on the communication scheme will be tested . Moreover , other preconditioners for CG-methods2 ; 13 ; 14 will be implemented . The goal of this investigation is to speed up the convergence of the CG-method and thereby to reduce the total execution time . Furthermore , it remains to examine if the described principles of the data distribution , the communication scheme and the overlapped execution apply successfully to other algorithms , such as the QMR-algorithm for solving non-hermitian linear systems of equations15 or Lanczos-methods for solving eigenproblems . Lanczosmethods are suited for determining eigenvalues and eigenvectors of dense and sparse matrices . In these methods , matrix-vector-products are computed and , depending on the special variant , linear systems of equations are solved . The last step is usually determining the eigenvalues of a tridiagonal matrix . A parallel algorithm for determining the eigenvalues of large real symmetric tridiagonal matrices has been already developed .16 Large eigenproblems occur in FE-models of mechanics , in particular dynamics and quantum mechanics , and in chemistry , especially in molecular dynamics and quantum chemistry . Fig. 6 . Speedups References 1 . M.R. Hestenes and E. Stiefel , Journal of Research of the National Bureau of Standards 49 -LRB- 1952 -RRB- , pp. 409 -LCB- 436 . 2 . J.M. Ortega , Introduction to Parallel and Vector Solution of Linear Systems , -LRB- Plenum Press , New York London , 1988 -RRB- . 3 . C. Aykanat , F. ? Ozg?uner , and D.S. Scott , Microprocessing and Microprogramming 29 -LRB- 1990 -RRB- , pp. 67 -LCB- 82 . 4 . G. T. Yeh , \ 3DFEMWATER : A Three-Dimensional Finite Element Model of Water Flow Through Saturated-Unsaturated Media '' , Oak Ridge National Laboratory ORNL-6386 , Aug. 1987 . 5 . H. Vereecken , G. Lindenmayr , A.Kuhr , D.H. Welte , and A. Basermann , \ Numerical Modelling of Field Scale Transport in Heterogeneous Variably Saturated Porous Media '' , KFA/ICG -4 Internal Report No . 500393 , Jan. 1993 . 6 . \ SMART , Benutzerhandb?ucher '' , Univ. . Stuttgart ISD-Berichte , 1976 -LCB- 1992 . 7 . C.P. Kruskal , L. Rudolph , and M. Snir , Theoretical Computer Science 64 -LRB- 1989 -RRB- , pp. 135 -LCB- 157 . 8 . S. Pissanetsky , Sparse Matrix Technology , -LRB- Academic Press , London Orlando , 1984 -RRB- . 9 . P. Fernandes and P. Girdinio , Parallel Computing 12 -LRB- 1989 -RRB- , pp . 327 -LCB- 333 . 10 . O.A. McBryan and E.F. Van de Velde , Parallel Computing 5 -LRB- 1987 -RRB- , pp. 117 -LCB- 125 . 11 . U. Schendel , Sparse -LCB- Matrizen , -LRB- R. Oldenbourg Verlag , M?unchen Wien , 1977 -RRB- . 12 . N. Feistl , \ Das Verfahren der konjugierten Gradienten auf Vektorrechern '' , LudwigMaximilians-Universit ? at M?unchen , Diplomarbeit , July 1987 . 13 . S.F. Ashby , SIAM J. Matrix Anal . Appl . 12 -LRB- 1991 -RRB- , pp. 766 -LCB- 789 . 14 . G. Pini and G. Gambolati , Adv. Water Resources 13 -LRB- 1990 -RRB- , pp . 147 -LCB- 153 . 15 . R.W. Freund and N.M. Nachtigal , Numerische Mathematik 60 -LRB- 1991 -RRB- , pp. 315 -LCB- 339 . 16 . A. Basermann and P. Weidner , Parallel Computing 18 -LRB- 1992 -RRB- , pp . 1129 -LCB- 1141 .

O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

communication scheme
conjugate gradients
data distribution
parallelization


Chapter 14 Comparing global optimization methods Juha Haataja Center for Scientific Computing P. O. Box 405 FIN-02101 Espoo , Finland E-mail : jhaataja@csc.fi Abstract . A selection of software packages for global optimization are compared using different types of test problems to illustrate the strengths and weaknesses of each method . The tested software packages use several different strategies for solving global optimization problems . A new software package called Gendo based on genetic algorithms and simulated annealing is introduced . Comparisons of this in Fortran 90 written package with other methods and packages are reported . A simple implementation of genetic annealing was found to be a promising approach , although more detailed analysis is needed . It seems that combining different types of methods -LRB- for example genetic algorithms and simulated annealing -RRB- may be a good strategy . Also , there is still work to be done in developing the methodology for comparing optimization methods and software . 14.1 Introduction Optimization is a frequent task in many fields of science and engineering . Increasing demands are posed for reliability and speed of the algorithms and software . The complete title of this article : A comparison of software for global optimization . 189 190 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Many real-world optimization problems are quite complicated , and the objective function may not be smooth or even continuous , which makes it very difficult to find solutions . An important finite-dimensional optimization problem is minimize x2Rn f -LRB- x -RRB- ; ai < = xi < = bi ; i = 1 ; : : : ; n ; -LRB- 14:1 -RRB- which has so-called box constraints . A local minimum x ? satisfies f -LRB- x ? -RRB- < = f -LRB- x -RRB- 8x 2 fx 2 Rn : kx ? x?k < = fflg : -LRB- 14:2 -RRB- The global minimum -LRB- if it exists -RRB- is the smallest of the local minima . Solving problem -LRB- 14.1 -RRB- can be very hard depending on the behaviour of the function f . If f is nonlinear , nonsmooth , and multimodal -LRB- e.g. it has several local minima -RRB- , any method can have difficulties to solve the optimization problem . For general combinatorial optimization problems there are several promising heuristic methods , such as simulated annealing , genetic algorithms , and tabu search -LSB- 274 , 440 , 441 -RSB- . For continuous variables the situation is much worse . Many methods perform extremely well for some test problems and fail miserably in others . Also , the needs of the user usually dictate what kind of method should be used . Sometimes only an approximate solution is needed , and sometimes we want to find the exact global minimizer with good precision and reliability . A good method for global optimization should not be trapped in a local minimum , and should find the approximate global optimum with a high degree of reliability . In addition to the performance and reliability of the method , its usefulness is determined by the class of problems it can be used to solve . A general-purpose method is a good starting point for a useful software package ; sometimes a modular approach of several different algorithms can be useful , however . Some users require \ black box '' solvers , others want to control every parameter of the method themselves . As such , many heuristic methods do n't -LRB- as yet -RRB- work well as black boxes | at least some experiments and tuning seems to be usually required . Adaptivity and parametric stability of the methods should be as important a factor in developing software as the performance of the package . And nowadays even a bad method can be successful on a fast computer . Recently there has been quite a lot of effort to compare different optimization packages and algorithms . Of course , some areas such as linear programming are already quite well established , but in the area of nonlinear programming much remains still to be done . In this article a few software packages are compared on a set of global optimization problems . One of the packages is written by the author ; the others are or have been widely used in global optimization . As such , this comparison is quite modest in scope and much additional work is needed . On the other hand , even this small experiment suggests a few possibilites for further research . Juha Haataja : Comparing global optimization methods . 191 14.2 Methods for global optimization Several different strategies for solving global optimization problems have been suggested -LRB- see for example -LSB- 294 , 660 -RSB- -RRB- . Solving real-world global optimization problems efficiently seems to require utilizing problem-specific information in the search strategy . Of course , this applies as well to smooth unimodal large-scale nonlinear optimization -LSB- 491 -RSB- . In global optimization the effect of a good search heuristic can be significant . On the other hand , an efficient heuristic for one problem can be very poor strategy for other problems . Perhaps the easiest way to approach global optimization is to take an efficient and realiable local optimization method -LRB- and a software implementation of it -RRB- , and add some simple multi-trial strategy to the basic algorithm . Local optimization with different starting points can be a good solution method , if the behaviour of the object function is well known , and good starting points can be generated . On the other hand , if the behaviour of the object function is not well known or can change radically depending on instance-specific parameters , then a more general optimization method is needed . Strategies utilizing for example search concentration -LSB- 1012 -RSB- , stochastic differential equations -LSB- 55 -RSB- , simulated annealing -LSB- 1334 -RSB- , genetic algorithms -LSB- 450 -RSB- , and tabu search -LSB- 440 , 441 -RSB- have been suggested . Many of these methods provide evidence for asymptotic convergence to the global optimum | but the performance penalty of increasing the probability of finding the global optimum can be high . Perhaps the worst problem in global optimization is \ knowing when to stop '' i.e. formulating good convergence criteria . A good overview of the field of the so-called evolutionary computation is given e.g. in -LSB- 1235 -RSB- . In this paper a software package using methods similar to both simulated annealing and genetic algorithms is compared with packages using other kinds of methods . It should be noted that there are many other implementations of both simulated annealing and genetic algorithms , and the results described here apply only to the -LRB- relatively simple -RRB- variants of the methods incorporated in the Gendo software package . 14.3 Examples of test problems for optimiza - tion To get some idea of the performance and reliability of algorithms and software for global optimization , several different types of problems were used in the tests . The object functions f1 , f2 , f3 and f4 are plotted in figure 14.1 . Function f5 is more like a real-world problem . Non-trivial cases of f5 have more than two variables , thus there is no illustration of the function . Test problem 1 : f1 -LRB- x -RRB- = nX i = 1 i x2i : 192 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 -2 -1 1 2 -2 2 5 10 15 f 1 -1 1 2 -2 -1 1 2 500 f 2 -5 5 -5 5 -10 10 20 f 3 -1 1 2 3 4 -1 1 2 3 4 5 10 f 4 Figure 14.1 : Two-dimensional versions of test functions f1 , f2 , f3 and f4 . These pictures illustrate the basic features of each test function . See section 14.3 for more information . Juha Haataja : Comparing global optimization methods . 193 Function f1 is convex and unimodal -LRB- see figure 14.1 -RRB- . The global minimum is at x ? = -LRB- 0 ; ; : : :-RRB- and the optimal value is f ? = 0 . Function f1 gets increasingly difficult to optimize as n grows due to the scaling by the term i . The feasible region used in the tests was fx 2 Rn : jxij < = 10g . Dimension n was set at 2 and 20 . Test problem 2 : f2 -LRB- x -RRB- = n = 2 X i = 1 100 -LRB- x2i ? x22i ? 1 -RRB- 2 + -LRB- 1 ? x2i ? 1 -RRB- 2 : Function f2 is non-convex but unimodal . The global minimum is at x ? = -LRB- 1 ; 1 ; : : :-RRB- and the optimum value is f ? = 0 . Function f2 is difficult to optimize because of the curving valley with very steep sides and gradual slope along the valley . The feasible region was fx 2 Rn : jxij < = 5g . Dimension n was set at 2 and 20 . Functions f1 and f2 can be minimized efficiently using software for unconstrained nonlinear optimization -LSB- 491 -RSB- . These problems were selected to test the methods on comparatively \ easy '' problems . Test problem 3 : f3 -LRB- x -RRB- = nX i = 1 x2i + 5 -LRB- 1 ? jxi + 1j -RRB- : Function f3 is non-convex , multimodal and nonsmooth . The global minimum is at x ? = -LRB- 2:5 ; 2:5 ; : : :-RRB- and the optimum value is f ? = ? 25n = 4 . There are 2n local optima , of which one is global . The feasible region was fx 2 Rn : jxij < = 10g . Dimension n was set at 2 and 20 . Test problem 4 : f4 -LRB- x -RRB- = nX i = 1 bj xi jc : Function f4 is non-convex and noncontinuous . The global minima are found in the set fx 2 Rn : jxij < 1g . The optimum value is f ? = 0 . The feasible region was fx 2 Rn : jxij < = 5:12 g. Dimension n was set at 2 and 20 . Test problem 5 : f5 -LRB- x -RRB- = kp -LRB- A -RRB- k jp -LRB->=-RRB- j : The constant matrix A of size q ? q has complex number elements , and scalar > = is also complex . Polynomial p is of degree m. To get a nontrivial problem q has to be at least m +1 . The coefficients of p are the variables of the optimization problem ; thus there are n = 2 -LRB- m + 1 -RRB- variables xi to optimize . Function f5 is in general non-smooth and multimodal . The feasible region was fx 2 Rn : jxij < = 100g . The degree of the polynomial p was set at m = 2 . The matrix A was chosen to be A = -LRB- aij -RRB- ; aij = i ? j p ? 1 ; i ; j 2 f1 ; 2 ; 3g : The constant > = was -LRB- 1 + p ? 1 -RRB- = 2 . 194 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Package Description References Gendo I genetic annealing -LSB- 274 , 1066 -RSB- Gendo II genetic algorithm -LSB- 274 , 450 -RSB- IMSL/UMPOL polytope method with a random starting point -LSB- 434 , 4 -RSB- Sigma stochastic differential equations -LSB- 55 -RSB- Smin1 clustering and local optimization -LSB- 1012 -RSB- Table 14.1 : Tested methods and software packages . The references provide basic information about the methods used in the software . 14.4 Software for global optimization Table 14.1 lists the tested software packages . Gendo is written in Fortran 90 by the author and provides facilities for studying two types of optimization methods and their hybridizations : genetic algorithms and simulated annealing -LSB- 274 -RSB- . The routine UMPOL from the IMSL Fortran Libraries is an implementation of the derivative-free optimization method known as the polytope method -LSB- 4 -RSB- . Sigma implements a method based on stochastic differential equations -LSB- 55 -RSB- . The package Smin1 uses a heuristic based on search-concentration methods -LSB- 1012 -RSB- . 14.4.1 Testing the software packages The test problems used in this article were reviewed in section 14.3 . The software packages are listed on table 14.1 . Two sizes of test problems f1 ; : : : ; f4 were used : n = 2 and n = 20 . All runs were repeated 12 times with different initial guesses to get statistical evidence of the convergence of the algorithms . All the tests were done on a Siligon Graphics workstation , which had the NAG Fortran 90 compiler in addition to the SGI Fortran 77 compiler . Run times of the packages are not reported here . Instead the number of function evaluations is used as the performance measure . This gives a better idea of the relative performance of each method . The convergence criteria were not very strict for UMPOL , Sigma , and Smin1 packages . In addition , the UMPOL package had an upper limit on the number of iterations . The Smin1 package includes many heuristic parameter , which were not tuned in the comparisons . Tuning would in fact be hard and not recommended , see ref . -LSB- 1012 -RSB- . The package can be used to make more than one optimization run | in the tests only one run was made . The Sigma package has a simple driver routine , and a more advanced calling syntax . In the tests the simple driver was not used -LRB- because of performance penalties -RRB- ; instead it was modified so that three independent optimization runs were made . In addition , the convergence tolerances were set to TOLREL = 10 ? 3 -LRB- relative tolerance -RRB- and TOLABS = 10 ? 6 -LRB- absolute tolerance -RRB- . The initial polytope in the UMPOL routine was random ; the size of the po - Juha Haataja : Comparing global optimization methods . 195 lytope was set up as -LRB- maxfbig ? minfaig -RRB- = 2 | see equation -LRB- 14.1 -RRB- . It should be noted that the UMPOL routine is intended for unconstrained optimization , so the upper and lower bounds were only used to set up the initial polytope . The genetic annealing method of Gendo package used a fixed number of iterations -LRB- 2000 for n = 2 and 20000 for n = 20 -RRB- . Population sizes were 10 -LRB- n = 2 -RRB- and 25 -LRB- n = 20 -RRB- . The tournament selection mechanism was used ; tournament sizes were 2 -LRB- n = 2 -RRB- and 6 -LRB- n = 20 -RRB- . Probability of crossover was 1 , and mutation rates were 0.33 and 0.033 . The box crossover -LRB- see section 14.5.3 -RRB- mechanism was used in the comparison tests . In the test problem f5 population size 10 and tournament size 2 were used . The setup of the genetic algorithm method of Gendo was similar to the genetic annealing method with some minor differences . Population sizes and the probabilities of mutation and crossover were the same . The tournament selection size 2 was used ; in addition , in the big test cases , elitism size 4 was used . This had similar effect as increasing the tournament size -LRB- speeding up convergence -RRB- . In addition to comparing Gendo with the other packages , the population size of the genetic annealing algorithm -LRB- see section 14.5.1 -RRB- in the Gendo software was changed between 10 and 100 to test the recombinative aspects of the method . The results are described in section 14.4.3 . In calculating the effects of the population size , uniform crossover with mixing was used . More detailed testing in this area remains to be done . 14.4.2 Results of the comparisons Table 14.2 gives results of the \ small '' tests -LRB- n = 2 -RRB- for functions f1 ; : : : ; f4 . Table 14.3 gives results for the \ big '' case n = 20 . Table 14.4 shows the performance of the methods for the test problem f5 -LRB- n = 6 -RRB- . In the tables f means the final optimum value reported by the software package ; f ? is the global optimum . Minimum and maximum values of the difference jf ? f?j in 12 runs are reported . Notation jf ? f?j means the average distance from the global optimum value calculated over 12 runs with different starting points . The test problems f1 and f2 illustrate how the global optimization packages Sigma and Smin use much more function evaluations than for example the IMSL routine UMPOL . See especially the results of f2 in the big test case -LRB- table 14.3 -RRB- . If a gradient-based method were used , the comparison would be even more discouraging . This shows the well-known fact that continuous unimodal functions are often very easy to optimize compared to real-world challenges in global optimization . Test problems f1 and f2 are hard for all the tested global optimization packages because the packages do n't have efficient local-search capabilities , and so the convergence is very slow although the approximate region is found quite efficiently . Although the Smin1 package has a polytope algorithm as a local optimizer , the 196 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Test function f1 , n = 2 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 0:0027 ; :024 -RSB- :011 -LSB- 2 000 ; 2 000 -RSB- 2 000 Gendo II -LSB- 0:0005 ; :042 -RSB- :012 -LSB- 2 000 ; 2 000 -RSB- 2 000 IMSL/UMPOL -LSB- 8:9 ? 10 ? 9 ; 2:3 ? 10 ? 8 -RSB- 1:6 ? 10 ? 8 -LSB- 68 ; 74 -RSB- 71 Sigma -LSB- 1:3 ? 10 ? 22 ; 6:1 ? 10 ? 12 -RSB- 1:4 ? 10 ? 12 -LSB- 11 000 ; 22 000 -RSB- 15 000 Smin1 -LSB- 3:8 ? 10 ? 15 ; 2 ? 10 ? 14 -RSB- 1:5 ? 10 ? 14 -LSB- 650 ; 680 -RSB- 670 Test function f2 , n = 2 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 0:015 ; :31 -RSB- :13 -LSB- 2 000 ; 2 000 -RSB- 2 000 Gendo II -LSB- 0:002 ; :031 -RSB- :014 -LSB- 2 000 ; 2 000 -RSB- 2 000 IMSL/UMPOL -LSB- 4:2 ? 10 ? 9 ; 2:3 ? 10 ? 8 -RSB- 1:2 ? 10 ? 8 -LSB- 77 ; 84 -RSB- 80 Sigma -LSB- 0 ; 4:7 ? 10 ? 8 -RSB- 4:5 ? 10 ? 9 -LSB- 21 000 ; 33 000 -RSB- 27 000 Smin1 -LSB- 0 ; 1:3 ? 10 ? 13 -RSB- 1:2 ? 10 ? 14 -LSB- 630 ; 700 -RSB- 670 Test function f3 , n = 2 , f ? = ? 12:5 . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 7:0 ? 10 ? 4 ; 2:7 ? 10 ? 2 -RSB- 8:8 ? 10 ? 3 -LSB- 2 000 ; 2 000 -RSB- 2 000 Gendo II -LSB- 3:8 ? 10 ? 4 ; 1:9 ? 10 ? 2 -RSB- 4:8 ? 10 ? 3 -LSB- 2 000 ; 2 000 -RSB- 2 000 IMSL/UMPOL -LSB- 0 ; 1:9 ? 10 ? 6 -RSB- 4:8 ? 10 ? 7 -LSB- 61 ; 66 -RSB- 63 Sigma -LSB- 0 ; 9:5 ? 10 ? 7 -RSB- 2:4 ? 10 ? 7 -LSB- 11 000 ; 17 000 -RSB- 14 000 Smin1 -LSB- 0 ; 9:5 ? 10 ? 7 -RSB- 1:6 ? 10 ? 7 -LSB- 2 900 ; 3 600 -RSB- 3 300 Test function f4 , n = 2 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 0 ; 0 -RSB- -LSB- 2 000 ; 2 000 -RSB- 2 000 Gendo II -LSB- 0 ; 0 -RSB- -LSB- 2 000 ; 2 000 -RSB- 2 000 IMSL/UMPOL -LSB- 0 ; 1 -RSB- :5 -LSB- 14 ; 17 -RSB- 15 Sigma -LSB- 0 ; 0 -RSB- -LSB- 12 000 ; 29 000 -RSB- 21 000 Smin1 -LSB- 0 ; 0 -RSB- -LSB- 1 300 ; 1 350 -RSB- 1 350 Table 14.2 : Results for test functions f1 , f2 , f3 and f4 when the dimension n was 2 . For each software package and test problem 12 independent test runs were performed . Table lists the maximum , minimum and average values of the object function reported by the routine . Also , maximum and minimum number of function avaluations is reported . See section 14.4.2 . Juha Haataja : Comparing global optimization methods . 197 behaviour on test problem f2 is seen to be quite poor . This is due to the high nonlinearity of the test problem -LSB- 491 , 434 -RSB- . Test problem f3 is multimodal and non-smooth , which poses problems for the Sigma package . Of course , the IMSL routine UMPOL tends to converge to the nearest local optimum . In this test problem only the Smin and Gendo packages were able to find the global optimum reliably in the case n = 20 . Test problem f4 is non-continuous , which posed great problems for UMPOL and Sigma packages . Of course , the polytope method of UMPOL is not meant for non-continous cases , and similarly the Sigma package expects the object function to be at least continuous . On the other hand , this problems was quite easy for both the Gendo and Smin1 packages . Test problems f5 was difficult to optimize precisely , although the Smin1 package did manage to find the minimum value quite robustly . On the other hand , the UMPOL routine of IMSL naturally converged to local optima , and the Sigma package did not always find the minimum with acceptable degree of precision . The Gendo package was slow to converge , but did give an estimate of the global optimum with some degree of precision . It should be noted that if the IMSL routine UMPOL had been used as a multistart routine with 12 runs , it would have been relatively successfull compared with the other methods . Test functions f2 and f4 were difficult to optimize realiably with UMPOL , though . 14.4.3 Effect of population size on genetic annealing Figure 14.2 shows how increasing the population size from 10 to 100 affects the genetic annealing algorithm . When using a small population size the algorithm tends to converge earlier | frequently to a non-global optimum . On the other hand , using a larger population size gives better results , and the probability of convergence to the global optimum is seen to increase . The comparisons presented in figure 14.2 were done using the test function f3 -LRB- n = 20 -RRB- , which is multimodal and nonsmooth . A more detailed analysis of the genetic annealing algorithm remains to be done using a set of different types of test problems . 14.5 Description of the Gendo package Gendo is an Fortran 90 code for global optimization written by the author of this article . Gendo includes implementations of genetic annealing and genetic algorithm type methods -LSB- 274 , 1066 -RSB- . Gendo is designed to be used as a subroutine library . Modularity makes it is easy to extend the software ; the module and type features of Fortran 90 were used extensively in developing the software . Numerical precision etc. details are implemented robustly using Fortran 90 features -LRB- see e.g. -LSB- 18 -RSB- -RRB- . Gendo has a simple trace facility for finding out what the program is doing . 198 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Test function f1 , n = 20 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 1:0 ; 4:5 -RSB- 2:4 -LSB- 20 000 ; 20 000 -RSB- 20 000 Gendo II -LSB- 2:0 ; 8:4 -RSB- 4:0 -LSB- 20 000 ; 20 000 -RSB- 20 000 IMSL -LSB- 4:9 ? 10 ? 8 ; 1:5 ? 10 ? 6 -RSB- 3:1 ? 10 ? 7 -LSB- 2 900 ; 4 500 -RSB- 3 700 Sigma -LSB- 6:5 ? 10 ? 7 ; 1:4 ? 10 ? 5 -RSB- 3:9 ? 10 ? 6 -LSB- 29 000 ; 39 000 -RSB- 36 000 Smin1 -LSB- 3:6 ? 10 ? 12 ; 2:1 ? 10 ? 9 -RSB- 5:7 ? 10 ? 10 -LSB- 32 000 ; 82 000 -RSB- 57 000 Test function f2 , n = 20 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 5:3 ; 46 -RSB- 24 -LSB- 20 000 ; 20 000 -RSB- 20 000 Gendo II -LSB- 17:0 ; 56 -RSB- 35 -LSB- 20 000 ; 20 000 -RSB- 20 000 IMSL -LSB- 0:046 ; 11:6 -RSB- 2:2 -LSB- 20 000 ; 20 000 -RSB- 20 000 Sigma -LSB- 7:1 ; 33 -RSB- 19 -LSB- 36 000 ; 38 000 -RSB- 38 000 Smin1 -LSB- 8:0 ? 10 ? 8 ; 2:0 ? 10 ? 7 -RSB- 1:4 ? 10 ? 7 -LSB- 150 000 ; 1 100 000 -RSB- 430 000 Test function f3 , n = 20 , f ? = ? 125 . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 0:060 ; :57 -RSB- :21 -LSB- 20 000 ; 20 000 -RSB- 20 000 Gendo II -LSB- 0:14 ; :93 -RSB- :43 -LSB- 20 000 ; 20 000 -RSB- 20 000 IMSL -LSB- 1:9 ? 10 ? 4 ; 90 -RSB- 18 -LSB- 20 000 ; 20 000 -RSB- 20 000 Sigma -LSB- 0:024 ; 20 -RSB- 6:8 -LSB- 18 000 ; 22 000 -RSB- 19 000 Smin1 -LSB- 0 ; 0 -RSB- -LSB- 98 000 ; 110 000 -RSB- 100 000 Test function f4 , n = 20 , f ? = . Routine jf ? f?j jf ? f?j f eval . average Gendo I -LSB- 0 ; 0 -RSB- -LSB- 20 000 ; 20 000 -RSB- 20 000 Gendo II -LSB- 0 ; 0 -RSB- -LSB- 20 000 ; 20 000 -RSB- 20 000 IMSL -LSB- 2 ; 23 -RSB- 10 -LSB- 20 000 ; 20 000 -RSB- 20 000 Sigma -LSB- 1 ; 22 -RSB- 11 -LSB- 16 000 ; 39 000 -RSB- 26 000 Smin1 -LSB- 0 ; 0 -RSB- -LSB- 4 200 ; 11 700 -RSB- 10 400 Table 14.3 : Results for test functions f1 , f2 , f3 and f4 for n = 20 . See also section 14.4.2 and table 14.2 . Juha Haataja : Comparing global optimization methods . 199 0.5 1 1.5 2 x 10 4 0.0001 0.001 0.01 0.1 1 10 100 Gendo I 100x200 f eval . 0.5 1 1.5 2 x 10 4 0.0001 0.001 0.01 0.1 1 10 100 Gendo I 25x800 f eval . 0.5 1 1.5 2 x 10 4 0.0001 0.001 0.01 0.1 1 10 100 Gendo I 10x2000 f eval . 0.5 1 1.5 2 x 10 4 0.0001 0.001 0.01 0.1 1 10 100 Gendo I 50x400 f eval . Figure 14.2 : The effect of the population size on the performance of the genetic annealing algorithm as implemented in the Gendo package . The horizontal axis gives the number of function evaluations , and the vertical axis give the residual jf ? f ? j. Test function f3 was used , dimension n was 20 . Population sizes were 100 , 50 , 25 , and 10 . Total number of function evaluations was set at 20 000 . For each population size 12 independent runs were made . The bold line shows the average value of the best individual , and the narrow lines represent the best and worst case results . Mutation probability was 0.033 for all runs . Uniform crossover was used with mixing probability 0.8 . 200 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Test function f5 , n = 6 , f ? ss :4428 . Routine f f f eval . average Gendo I -LSB- 0:4438 ; :4558 -RSB- :4470 -LSB- 20 000 ; 20 000 -RSB- 20 000 Gendo II -LSB- 0:4458 ; :4696 -RSB- :4540 -LSB- 20 000 ; 20 000 -RSB- 20 000 IMSL -LSB- 0:4428 ; :5451 -RSB- :4520 -LSB- 600 ; 600 -RSB- 600 Sigma -LSB- 0:4506 ; :4820 -RSB- :4651 -LSB- 20 000 ; 35 000 -RSB- 27 000 Smin1 -LSB- 0:4428 ; :4428 -RSB- :4428 -LSB- 20 400 ; 40 900 -RSB- 28 400 Table 14.4 : Results for test function f5 for n = 6 . See section 14.4.2 . There are also facilities for analysis of the algorithms . Some implementation details of Gendo are described in section 14.6 . Sections 14.5.1 and 14.5.2 describe the basic algorithms implemented in the Gendo package . Results of comparisons with other software packages were described in section 14.4.2 . 14.5.1 Genetic annealing in global optimization Genetic annealing is a method which has adopted some features of genetic algorithms and simulated annealing -LSB- 1066 -RSB- . Genetic annealing is a parallel search strategy utilizing a population of solution candidates . Various recombination operators can be used similarly to genetic algorithms -LRB- see sections 14.5.2 and 14.5.3 -RRB- . Selection of new candidates is based on the concepts of \ threshold '' and \ energy storage '' . Depending on the parameters the method is either a random search method , a genetic algorithm type method , or an annealing method . Performance of the algorithm could be improved by adding local optimization , niching etc. . Here is a general outline of the algorithm as implemented in Gendo : Algorithm 1 Genetic annealing Initialize population fxig ; i = 1 ; : : : ; l. Initialize thresholds Ti with values of the object function : Ti = f -LRB- xi -RRB- . Select the cooling constant c 2 -LRB- 0 ; 1 -RRB- . Step A : Empty the energy bank : E = 0 . for i = 1 ; 2 ; : : : ; l Recombine xi with a selected element : xi ! x0 . Generate mutations : x0 ! x00 . Evaluate the object function : fi = f -LRB- x00 -RRB- . if fi > Ti Keep the old element xi . Juha Haataja : Comparing global optimization methods . 201 else Set E = E + Ti ? fi and Ti = fi . Replace xi with the new element x00 . endif end Set Ec = cE = l and Ti = Ti + Ec ; i = 1 ; : : : ; l. Return to step A . The genetic annealing algorithm used here is based on an algorithm for combinatorial optimization -LSB- 1066 -RSB- . The coding of the search space was done by using real-valued variables . The selection and recombination operators are described in section 14.5.3 . 14.5.2 A genetic algorithm for global optimization Genetic algorithms have been used to solve global optimization problems . Here a basic version of genetic algorithms is used , which does not mean that there are no more efficient versions of this method available . More detailed comparisons and analysis of genetic algorithms and genetic annealing are needed to see what kind of performance and reliability the methods can give . Algorithm 2 Simple genetic algorithm Initialize population fxig ; i = 1 ; : : : ; l. Calculate the values of the object function : f -LRB- xi -RRB- . for j = 1 ; 2 ; : : : ; n for i = 1 ; 2 ; : : : ; l Recombine two selected elements xa and xb to generate x0 . Generate mutations : x0 ! x00 . Evaluate the object function : fi = f -LRB- x00 -RRB- . Store the new element in the new population . end end The genetic algorithm described above and used in Gendo is a simple variant of the classical genetic algorithm -LSB- 450 -RSB- . Gendo does not use binary coding -LRB- allthough implementing it would be straightforward using the existing modules and variable types -RRB- . Instead , real coding and real recombination operators are used . 14.5.3 Selection and recombination operators The Gendo package includes implementations of several different selection and recombination mechanisms . In this article in all tests the tournament selection mechanism was used , due to its simplicity and lack of need to provide a scheme for the scaling of the values of the object function . 202 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 Several different recombination strategies are included in the Gendo package , but in the tests only the uniform mixing mechanism -LRB- described below -RRB- was used . In the following real-coded parameter vectors are denoted by x and y 2 Rn . The vector x has n components xi . Box crossover recombines two parents x1 and x2 producing offspring of the type y 2 Rn ; yi = rix1i + -LRB- 1 ? r1 -RRB- x2i ; where r is a vector whose components are selected from the uniform random distribution so that ri 2 -LSB- 0 ; 1 -RSB- . Line crossover is similar to box crossover , but instead of a vector of random variables there is a scalar r 2 -LSB- 0 ; 1 -RSB- : y 2 Rn ; yi = rx1i + -LRB- 1 ? r -RRB- x2i : In convex crossover we have p parents -LRB- p can vary from iteration to iteration -RRB- . The recombination produces an offspring by forming a random convex combination of the parents : y 2 Rn ; yi = pX j = 1 rjxji ; where the vector r of p real random numbers 2 -LSB- 0 ; 1 -RSB- is normalized so that Ppj = 1 rj = 1 . The Gendo package also includes implementations of one-point , two-point and uniform crossover . Because of using real coding , the mixing of the component values of the code vectors is done using a probabilistic mechanism . In the one-point crossover the offspring is formed by selecting the crossover point i 2 f1 ; 2 ; : : : ; ng and copying the first i ? 1 components of parent x1 to the offspring ; the other components yi ; yi +1 ; : : : ; yn are taken from parent x2 . The component yi is taken to be rx1i + -LRB- 1 ? r -RRB- x2i ; r 2 -LSB- 0 ; 1 -RSB- with a given probability P . If P = no mixing occurs at the crossover point ; if P = 1 the component yi is always a combination of the corresponding components in the parents . The mixing approach can easily be extended to two-point and uniform crossover . In the comparison tests reported in tables 14.2 to 14.4 the box crossover was used . In testing the effects of the population size uniform crossover with mixing probability P = :8 was used . Both of these mechanism provide good recombinative properties at least for the selected test problem set . 14.6 Implementing Gendo using Fortran 90 Fortran 90 is the new standard of the Fortran programming language . It still has as a subset almost all of the Fortran 77 standard , although some features will Juha Haataja : Comparing global optimization methods . 203 be removed in the near future . Fortran 90 supports modularity but is not an object-oriented language in the strict sense . Fortran 90 is well suited for developing software for numerical calculations in science and engineering . Some of the key features of Fortran 90 are : ffl a wide selection of control structures , ffl different levels of numerical precision supported , ffl multi-dimensional array syntax and operators , ffl support for complex arithmetic , ffl modules , new data types , and operator overloading , ffl good facilities for dynamic memory allocation , ffl extensive compile-time error checking . The experiences in implementing Gendo suggest that Fortran 90 compilers are already sufficiently robust and efficient for practical work . Many hardware vendors -LRB- such as Cray -RRB- have announced native Fortran 90 compilers . This will mean increasing performance and reliability on these machines . The NAG Fortran 90 compiler was used in the tests reported in this article . In addition , the Gendo package was compiled using the IBM Fortran 90 compiler to check the portability of the code . Some inconsistensies in the interpretation of the Fortran 90 standard were discovered , but after taking the strict view the compilation posed no difficulties . 14.7 Suggestions for future work The package Gendo is intended to be used as a testbed of heuristic approaches to global optimization . Some plans for further research are : ffl including more demanding test problems in the test set , ffl testing different kinds of adaptive convergence criteria , ffl effect of local optimization as a component of the methods , ffl comparing different versions of genetic algorithms and genetic annealing , ffl implementing parallel processing using message-passing libraries -LRB- such as PVM -RRB- . In addition , there should probably be more possibilities for user control of the methods without recompiling the modules . Also , more tools for analyzing the dynamic behaviour of the algorithms should be incorporated in the software . 204 Proceedings of the 1NWGA , Vaasa , Finland , 9 - 12 January 1995 14.8 Conclusions This article presented some results of comparing different approaches to global optimization on a small set of test problems . A simple implementation of genetic annealing was found to be a promising approach , although more detailed analysis is needed . It seems that combining different types of methods -LRB- for example genetic algorithms and simulated annealing -RRB- may be a good strategy . Also , there is still work to be done in developing the methodology for comparing optimization methods and software . Standard test problem sets and environments are becoming widely available ; these should be used in future comparisons . Finally , I believe that implementing numerical algorithms in Fortran 90 is a good idea . Extensive compile-time error checking , modularity , and good support for numerical datatypes makes it ideal for this kind of work . Compared to C++ it is of course apparent that Fortran 90 falls short of the object-oriented approach . On the other hand , C++ is perhaps too complex and dynamic language to be used effectively by the numerically oriented scientific community . 14.9 Acknowledgements I thank Jarmo Alander for useful discussions and for prompting me to finish this article on time . Lii Xiaoshu from the Helsinki University of Technology gave me the idea to use the test problem f5 in the comparisons . Finally , I thank my colleagues at CSC , especially Jussi Rahola , Juhani K?apyaho , and Juha Fagerholm , for making it possible to educate myself even though the workload has been often pressing . 14.10 References The CSC guidebook -LSB- 489 -RSB- is an introduction to solving optimization problems written in Finnish -LRB- see also -LSB- 490 -RSB- -RRB- . The Fortran 90 language is described in the references -LSB- 18 -RSB- . Some methods for global optimization are reviewed in the article -LSB- 293 -RSB- . Simulated annealing is described in the books -LSB- 274 , 1334 -RSB- , and genetic algorithms in the book -LSB- 450 -RSB- , among others . 14.11 Bibliography See chapter bibliography at the end of this proceedings .

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O B I O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I B I O O O O O O O O O B I O B I O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O

fortran 90
genetic algorithms
genetic annealing
global optimization
simulated annealing