Turing's Paper on Rounding-Off Errors in Matrix Processes

Before 1940, solving a set of n linear equations with n ≥ 10 was regarded as a very challenging problem. Several direct methods were used for solving linear equations, but only during the 1940s did mathematicians realise that most of those direct methods were closely related to Gausian elimination, and also to the factorisation of a square matrix as a product of lower and upper triangular matrices. Some linear problems in physics and engineering had been solved by indirect methods of successive approximation, which later were systematised as various iterative methods. As more powerful calculating machines were developed, some scientists began solving larger sets of linear equations. But serious concern was felt about the effects of round-off in the computation, which might produce large errors in the computed solution for n > 10 (von Neumann & Goldstine [5, footnote 11]). At a conference in 1941, Harold Hotelling argued (Hotelling [2]) that in solving a set of n equations the calculation should be done with n log10 4 extra figures — for each type of machine that implied severe limitations on the size of linear systems which could be solved. In 1947, John von Neumann and Herman H Goldstine published their very detailed study of “Numerical inverting of matrices of high order”, in which they concentrated mainly on inversion of symmetric positive-definite matrices. They expressed upper bounds for the error of the computed inverse in terms of “various quantities and properties which cannot be supposed to be known when the problem of inverting a matrix A or AI comes up”, including the maximum and minimum moduli of eigenvalues of the matrix (von Neumann & Goldstine [5, p. 1089]). They anticipated that, when electronic digital computers become available for inverting matrices, then n ∼ 100 will become manageable (von Neumann & Goldstine [5, p. 1031]). They remarked that “An approximate inverse of a matrix P might be defined as one which lies close to the exact inverse P −1 . From the point of view of numerical procedure it seems more appropriate, however, to interpret it as the inverse P′−1 of a matrix P′ that lies close to P that is, to permit an uncertainty of, say, ǫ in every element of P”. (von Neumann & Goldstine [5, p. 1092]). That seems to be the earliest published mention of backward error analysis. In 1948, Alan Mathison Turing published his influential paper on “Rounding-off errors in matrix processes” (Turing [4]). At the National Physical Laboratory he had collaborated with Leslie Fox, Harry D Huskey and James H Wilkinson in extensive tests of the elimination method on many sets of linear equations (Fox, Huskey & Wilkinson [1]). He reported that “Fox found that no exponential build-up of errors such as that envisaged by Hotelling actually occurred. In the meantime another theoretical investigation was being carried out by J von Neumann, who reached conclusions similar to those of this paper for the case of positive definite matrices, and communicated them to the writer at Princeton in January 1947 before the proofs given here were complete” (Turing [4, p. 288]). If a set of linear equations Ax = b is to be solved for a single vector b then the computational cost is about one-third of that for inverting A and then evaluating x as A−1b; but if solutions are required for several vectors b then A should be inverted. Turing considered that “It seems probable that with the advent of electronic computers it will become standard practice to find the inverse”. However, good computing practice still uses solution (and not inversion) for the usual case of a single vector b. Turing gave a constructive proof that if the principal minors of the matrix A are non-singular then A factorises as A = LDU uniquely, where L is a unit lower triangle, D is a diagonal matrix and U is a unit upper triangle. (Likewise A = U′D′L′ uniquely, where L′ is a unit Turin ’ r n Rounding-Off Errors in Matrix Processes Garry J Tee