Jacobi Iteration and Spectral Radius | by Ryan Reiff | Towards Data Science

One of the main pillars of Numerical Analysis is the solving of large linear systems of equations. There are a variety of methods that Numerical Analysts implement in order to solve such systems; however, the one we will look at today is Jacobi Iteration.
Problem Statement:
A large linear system can easily be represented with matrices in the form “Ax=b”, where “A” represents a square matrix that contains the ordered coefficients of our linear system of equations, “x” holds all of our different variables, and “b” represents the constants that each linear equation is equal to. We wish to solve for our unknown x-values, and we can do so through the use of Jacobi Iteration.
In order to fully understand Jacobi Iteration, we must first understand Fixed Point Iteration. Both methods utilize the same scheme, but Jacobi Iteration can be applied to a larger system of equations. In Fixed Point Iteration, the main idea is to take an equation and arrange it in terms of Xn+1 = F(Xn), so that by starting at some initial x-value (Xn) and plugging it into the F(Xn) equation, we get a new value (Xn+1) that we then use as the next x-value to plug into F(Xn), and so on and so on. We can repeat this process until the two sides of the equation become equal or roughly equal, in which case we have reached our fixed point solution. A small worked out example is shown below. This method does not always converge and there are certain tests to determine if it will; however, we will just stick with this simple explanation to summarize the main idea for now.
Now, let’s take a look at the way Jacobi Iteration leverages the principles of Fixed Point Iteration in the example below.
Example System of Equations:
With Jacobi Iteration, just like with normal Fixed Point Iteration, we are interested in taking an equation and rearranging it so that it takes the form Xn+1 = F(Xn). The only difference is that with Jacobi iteration we are doing this to not just one equation, but rather every equation in our system of equations, so that each equation is equal to a single unique variable (one equation equal to x1, another for x2, and so on). This reorganization of equations is shown below.
To iterate through this system, we can start with a set of initial values for X1,X2, and X3 and plug these values into our equations. We can then use our new values and plug them back into the equation once more. Repeat this process, as shown below, for the first few iterations.
Assuming that our system of equations converges using Jacobi Iteration, if we continue this process, eventually we will converge upon our solution or get reasonably close up to a given tolerance. While conceptually the process is quite simple, there is a bit of nuance involved when it comes to checking to see if convergence is actually possible.
The Formal Jacobi Iteration Equation:
The Jacobi Iterative Method can be summarized with the equation below. The “a” variables represent the elements of the coefficient matrix “A”, the “x” variables represent our unknown x-values that we are solving for, and “b” represents the constants of each equation.
By applying the Jacobi Iteration Equation above, we are left with the same conclusion that we conceptually derived earlier when discussing the relationship between Fixed Point Iteration and Jacobi Iteration.
What is the “T” Matrix? Why is it important?:
While the implementation of the Jacobi iteration is very simple, the method will not always converge to a set of solutions. Due to this fact, a convergence test must be run prior to implementing the Jacobi Iteration. This convergence test is entirely dependent on a new matrix called our “T” matrix. Take the matrix representation of our Jacobi Iteration.
We can re-write this system of equations in such a way that the entire system is decomposed into the form “Xn+1 = TXn + c”. In other words, we can decompose the matrix on the right hand side of the equation into a matrix of coefficients and into a matrix of constants. The process is shown below. It will be shown later why getting our system of equations into this form (Xn+1=TXn + c) is crucial in order to test for convergence.
For our specific example, our Jacobi Iteration Matrix decomposed into this form will be exactly equivalent to
Where:
While the derivation above is very useful in understanding where the matrix “T” comes from, it is not necessary to do every time in order to actually find out what the matrix “T” is. Rather, the steps below can be used to fill in every element of the “T” matrix in a far more simple and timely fashion.
The “T” matrix is extremely important because all that is required for our Jacobi Iteration Method to converge, is that the spectral radius of our matrix “T” is strictly less than 1.
The spectral radius for a square matrix is defined simply as the largest absolute value of its eigenvalues. The proof below demonstrates why it is so crucial that we solve for matrix “T” in the first place, and how it’s relationship to the spectral radius creates the condition that the spectral radius must be less than 1 if we wish to see our method converge.
Test of Convergence for Jacobi Iteration:
Now that we know how to test for convergence (and more importantly why we use this certain test), we can do so with our example system of equations. We have already solved for our “T” matrix from earlier, so all that is left to do is to find all of its eigenvalues and make sure their absolute values are strictly less than one.
The eigenvalues for the “T” matrix in our example are listed below.
Since all of their absolute values are less than 1, our Jacobi Iteration Method will converge, and all that is left to do is implement some Python code that runs the iterations for us.
Implementation:
import mathimport numpy as npdef JacobiIteration(n):    x1 = np.zeros(n)    x2 = np.zeros(n)    x3 = np.zeros(n)        for i in range(0,n-1):        x1[i+1] = -x2[i] - 0.5*x3[i] + 3        x2[i+1] = 0.5*x1[i] - 0.5*x3[i] + 1.5        x3[i+1] = -0.5*x1[i] - x2[i] + 3.5    print ("Our x1 value is",x1[-1],           "\nOur x2 value is",x2[-1],           "\nOur x3 value is",x3[-1])JacobiIteration(60)
After about 60 iterations, and starting from an initial guess of zero for all of our x-values, we can see that we have converged just about exactly upon our solution given the machine precision of the computer.
In the image on the left, we can see the output from each iteration. This is only about 30 iterations, and we are still within an extremely high degree of accuracy. The order of the arrays from top to bottom is x1, x2, then x3. They read row by row, left to right.
That’s really all there is to Jacobi Iteration. We take a system of equations, rearrange it a bit, test for convergence, run a bit of code, and then we are done.
Citations:
Burden, Richard L., and J. Douglas. Faires. Numerical Analysis, by Richard L. Burden, J. Douglas Faires. 9th ed., Brooks/Cole, 2010. pp. 452
Hector D. Ceniceros, 2020, Chapter 10.9: Convergence of Linear Iterative Methods, lecture notes, Numerical Analysis 104B, University of California Santa Barbara, delivered February 2020.