A Block Orthogonalization Procedure with Constant Synchronization Requirements

A BLOCK ORTHOGONALIZATION PROCEDURE WITH CONSTANT SYNCHRONIZATION REQUIREMENTS ANDREAS STATHOPOULOS AND KESHENG WU y Abstract. We propose an alternative orthonormalization method that computes the orthonor- mal basis from the right singular vectors of a matrix. Its advantage are: a all operations are matrix-matrix multiplications and thus cache-e cient, b only one synchronization point is required in parallel implementations, c could be more stable than Gram-Schmidt. In addition, we consider the problem of incremental orthonormalization where a block of vectors is orthonormalized against a previously orthonormal set of vectors and among itself. We solve this problem by alternating itera- tively between a phase of Gram-Schmidt and a phase of the new method. We provide error analysis and use it to derive bounds on how accurately the two successive orthonormalization phases should be performed to minimize total work performed. Our experiments con rm the favorable numerical behavior of the new method and its e ectiveness on modern parallel computers. Key words. Gram-Schmidt, orthogonalization, Householder, QR factorization, singular value decomposition, Poincare AMS Subject Classi cation. 65F15 1. Introduction. Computing an orthonormal basis from a given set of vectors is a basic computation, common in most scienti c applications. Often, it is also one of the most computationally demanding procedures because the vectors are of large dimension, and because the computation scales as the square of the number of vectors involved. Further, among several orthonormalization techniques the ones that ensure high accuracy are the more expensive ones. In many applications, orthonormalization occurs in an incremental fashion, where a new set of vectors we call this internal set is orthogonalized against a previously orthonormal set of vectors we call this external, and then among themselves. This computation is typical in block Krylov methods, where the Krylov basis is expanded by a block of vectors 12, 11 . It is also typical when certain external orthogonalization constraints have to be applied to the vectors of an iterative method. Locking of converged eigenvectors in eigenvalue iterative methods is such an example 19, 22 . This problem di ers from the classical QR factorization in that the external set of vectors should not be modi ed. Therefore, a two phase process is required; rst orthogonalizing the internal vectors against the external, and second the internal among themselves. Usually, the number of the internal vectors is much smaller than the external ones, and signi cantly smaller than their dimension. Another important di erence is that the accuracy of the R matrix of the QR factorization is not of pri- mary interest, but rather the orthonormality of the produced vectors Q . A variety of orthogonalization techniques exist for both phases. For the external phase, Gram- Schmidt GS and its modi ed version MGS are the most competitive choices. For the internal phase, QR factorization using Householder transformations is the most stable, albeit more expensive method 11 . When the number of vectors is signi - cantly smaller than their dimension, MGS or GS with reorthogonalization are usually preferred. Computationally, MGS, GS and Housholder transformations are based on level 1 or level 2 BLAS operations 15, 9, 8 . These basic kernel computations, dot prod- ucts, vector updates and sometimes matrix-vector operations, cannot fully utilize the Department of Computer Science, College of William and Mary, Williamsburg, Virginia 23187- 8795, andreas@cs.wm. edu . y NERSC, Lawrence Berkeley National Laboratory, Berkeley, California 94720, kwu@lbl.gov .