Complete Step-by-step Conjugate Gradient Algorithm from Scratch | by Albers Uzila | Towards Data Science

Table of Contents (read till the end to see how you can get the complete python code of this story)· Conjugate Gradient for Solving a Linear System· Improving the Algorithm  ∘ Theorem  ∘ Simple Problems· Conjugate Gradient for Nonlinear Optimization Problem  ∘ Wolfe Line Search  ∘ Implementation  ∘ Scenarios· Conclusion
The Conjugate Gradient is one of those algorithms that are mathematically involved. So, to simplify things, we will cut off any unnecessary and long explanation. Later, we will also use a math theorem but won’t prove it. Let’s import some libraries first.
Consider a linear equation Ax = b where A is an n × n symmetric positive definite matrix, x and b are n × 1 vectors. To solve this equation for x is equivalent to a minimization problem of a convex function f(x) below
that is, both of these problems have the same unique solution.
We will solve Ax = b for x iteratively. Define residual r(x) := ∇f(x) = Ax − b. Then at iterate xk, we have
The one-dimensional minimizer of f along xk + αpk is given explicitly (by setting the first derivative of f(xk + αpk) w.r.t. α to be zero) as follows
In the conjugate gradient method, the direction set {p0, p1, ...} is chosen such that its elements are conjugates with respect to A, that is,
Each direction pk is chosen to be a linear combination of the negative residual -rk (which is the steepest descent direction for the function f) and the previous direction pk−1. We write
where the scalar βk is to be determined by the requirement that pk−1 and pk must be conjugate with respect to A. By pre-multiplying the last equation by
and imposing the condition
we find that
Knowing αk and pk, the next iterate could be calculated as usual using
As previously mentioned, the direction set {p0, p1, ...} is chosen such that its elements are conjugates with respect to A. But how do we choose the first search direction p0? Simply use the good ol’ steepest descent direction at the initial point x0, that is, p0 = -r0.
We now have a working algorithm:
1. Initialize x02. Calculate r0 = Ax0 − b3. Assign p0 = −r04. For k = 0, 1, 2, ...:    * calculate αk = -rk'pk / pk'Apk    * update xk+1 = xk + αkpk    * calculate rk+1 = Axk+1 - b    * calculate βk+1 = rk+1'Apk / pk'Apk    * update pk+1 = -rk+1 + βk+1pk
Here, the prime notation denotes the transpose operation for writing convenience.
towardsdatascience.com
We could further improve the time and space complexity in calculating αk and βk. To do that, we put forward the following theorem. As said before, we won’t prove the theorem here.
Let x0 ∈ Rn be any starting point and suppose that the sequence {xk} is generated as explained above. Then
Fix k > 0. Since we have
then
and hence
Since by Theorem 1 the right-hand side is zero, we then have
Divide both sides by
we obtain
The left-hand side of the last equation is precisely αk. So we have a more economical form of αk as follows
Since rk = Axk − b, we have rk+1 − rk = A(xk+1 − xk). Combined with the step equation xk+1 = xk + αkpk, we obtain
Substitute the last equation to the established βk+1 equation above, we got
Consider the following observations:
From Theorem 1, we have
We proved previously that
Since
we have
hence
Since by Theorem 1
we got
Substitute these three observations to the latest βk+1 equation above, we obtain
Now we have all the ingredients to build the conjugate gradient algorithm for solving linear systems.
We will try to use this algorithm to solve Ax = b for x, where A and b are defined differently for the three problems below.
A is a 2 × 2 symmetric positive-definite matrix and b is a 2 × 1 vector. Both are generated randomly using make_spd_matrix function from scikit-learn .
A [[ 2.5409 -0.0113]  [-0.0113  0.5287]]b [1.3864  0.3719]The solution x* should be [0.5488  0.7152]
Let’s run the conjugate gradient algorithm with the initial point x0 at [-3, -4].
Iteration: 1 	 x = [ 0.7428 -2.9758] 	 residual = 2.0254Iteration: 2 	 x = [ 0.5488  0.7152] 	 residual = 0.0000Solution: 	 x = [ 0.5488  0.7152]
The solution is found in two iterations. We could check whether the solution is the desired one using allclose function from numpy . If it returns True , the solution is correct.
True
Lastly, we can turn this problem into a minimization of a function f, where f is defined below.
To visualize the learning path, we create some python functions as follows.
From the plot above, we can confirm that the solution is indeed found in two iterations. For the other two problems below, we won’t bother plotting since the dimension of x is more than 2 and hence it’s hard or impossible to plot.
A is a 3 × 3 symmetric positive-definite matrix and b is a 3 × 1 vector, both are generated randomly as before.
A [[ 0.7444 -0.5055 -0.0851]  [-0.5055  3.4858  0.0572]  [-0.0851  0.0572  0.4738]]b [-0.0043  2.2501  0.2798]The solution x* should be [ 0.5488  0.7152  0.6028]
Let’s run the conjugate gradient algorithm with the initial point at [3, 1, -7].
Iteration: 1 	 x = [ 0.0261  1.8702 -2.1522] 	 residual = 4.3649Iteration: 2 	 x = [-0.5372  0.5115 -0.3009] 	 residual = 0.7490Iteration: 3 	 x = [ 0.5488  0.7152  0.6028] 	 residual = 0.0000Solution: 	 x = [ 0.5488  0.7152  0.6028]
The solution is found in three iterations. Again, we see that this is indeed the solution we want.
True
A is a 6 × 6 symmetric positive-definite matrix and b is a 6 × 1 vector, both are generated randomly as before.
A [[ 3.4430 -0.3963  2.5012  0.9525  0.6084 -1.2728]  [-0.3963  0.6015 -0.4108 -0.1359 -0.0295  0.2630]  [ 2.5012 -0.4108  2.5927  0.7072  0.5587 -1.0613]  [ 0.9525 -0.1359  0.7072  1.1634  0.1920 -0.4344]  [ 0.6084 -0.0295  0.5587  0.1920  0.7636 -0.3261]  [-1.2728  0.2630 -1.0613 -0.4344 -0.3261  1.0869]]b [ 3.0685  0.0484  2.5783  1.2865  0.8671 -0.8230]The solution x* should be [ 0.5488  0.7152  0.6028  0.5449  0.4237  0.6459]
Let’s run the conjugate gradient algorithm with the initial point at [9, 0, -2, 3, -2, 5].
Iteration: 1   x = [ 6.0798  0.2932 -3.6765  1.9076 -2.1460  5.5981]               residual = 5.2930Iteration: 2   x = [ 1.5966 -0.3208 -0.2777 -0.5521  1.3176  1.3223]               residual = 1.1933Iteration: 3   x = [ 0.6074  0.2616  0.2860  0.7581  0.8068  0.6812]               residual = 0.3795Iteration: 4   x = [ 0.5343  0.6703  0.6294  0.5496  0.4109  0.6765]               residual = 0.0346Iteration: 5   x = [ 0.5477  0.7172  0.6058  0.5464  0.4280  0.6519]               residual = 0.0047Iteration: 6   x = [ 0.5488  0.7152  0.6028  0.5449  0.4237  0.6459]               residual = 0.0000Solution:      x = [ 0.5488  0.7152  0.6028  0.5449  0.4237  0.6459]The solution is correct.
The solution is found in six iterations. Again, we see that this is indeed the solution we want.
What can we learn from these examples? The most obvious one is that the iteration needed for the conjugate gradient algorithm to find the solution is the same as the dimension of matrix A. That’s why we don’t need to safeguard our algorithm from infinite loop (using max iteration for instance) in LinearCG function. In fact, if A has only r distinct eigenvalues, then the Conjugate Gradient iteration will terminate at the solution in at most r iterations.
The Conjugate Gradient method is recommended only for large problems; otherwise, Gaussian elimination or other factorization algorithms such as the singular value decomposition are to be preferred, since they are less sensitive to rounding errors.
In the previous section, we’ve implemented Conjugate Gradient as a minimization algorithm for the convex quadratic function f. We are ready to step up our game. Can we adapt the approach to minimize general convex functions or even general nonlinear functions f?
We can do this by changing two things in the previous algorithm:
The steplength αk is currently the exact minimizer of f along pk. Instead, we would like to have αk as the approximation of the minimizer using a line search algorithm.
The residual rk now should be the gradient of the objective function f.
We’ve introduced a line search algorithm in the previous article: Armijo Line Search. However, since in the conjugate gradient method now we have
the sufficient decrease condition alone is not enough to ensure descent direction if βkpk−1 becomes dominant.
Enter Wolfe Line Search.
We add another condition, called the curvature condition, which requires αk to satisfy
for some constant c2 ∈ (c1, 1) where c1 is the constant from the Armijo condition. Note that the left-hand side is simply the derivative of f(xk + αpk) at αk, so the curvature condition ensures that the slope of f(xk + αpk) at αk is greater than c2 times the initial slope
This condition rules out unacceptably short steps preferred by the Armijo condition and hence f could be reduced significantly. Combined with the Armijo condition, the two conditions are also called the Wolfe condition.
We can modify the curvature condition to force αk to lie in at least a broad neighborhood of a local minimizer or stationary point of f. This modification is called the strong Wolfe condition and require αk to satisfy
For the Armijo Line Search that we discussed in the previous article, we locate the final step length by multiplying αk with a shrinking term ρ. This strategy will be inefficient to be implemented for the strong Wolfe condition since it defeats the purpose of the condition. So, for the selection phase, we zoom in to locate the final step length, which usually reduces the bracketing interval during its search for the desired step length and interpolates some of the function and derivative information gathered on earlier steps to guess the location of the minimizer. We would not explain the math here since it’s very tedious and it may be better to directly read the code below.
Wow, this is a long one. Now we are ready to implement Wolfe Line Search to Conjugate Gradient Algorithm.
First of all, let’s define an objective function to solve: the Griewank Function.
We will be using the two-dimensional version of the Griewank Function (n = 2) as done in the previous article.
Finally, let’s build the nonlinear Conjugate Gradient algorithm. As stated before, we use
as the direction and αk from Wolfe Line Search as the step length. Also, steps will be taken until one of the stopping criteria below is satisfied:
The norm of the gradient of f is close enough to zero, that is,
The number of steps taken is 1000
The beforementioned formula for βk is developed by Fletcher-Reeves (FR). One main problem with this method is that if we, for any reason, end up in a bad direction pk then FR continues to generate poor directions. For this reason, Polak-Ribiere (PR) developed another form of βk,
The Conjugate Gradient algorithm with PR often works better than FR in practice. However, the Strong Wolfe condition does not guarantee that pk is a descent direction.
In the code below, we also introduce many other variations of βk:
Hestenes-Stiefel (HS)
Dai-Yuan (DY)
Hager-Zhang (HZ)
We then make a python function for creating two plots:
learning path of x along with the contour plot of f(x)
the value of f(x) per step taken
For an initial value x0 and some β estimation method, we run Conjugate Gradient algorithm on f with different scenarios:
x0 = [0, 3] and use FR
x0 = [2, 1] and use FR
x0 = [2, 1] and use PR
x0 = [2, 1] and use HS
x0 = [2, 1] and use DY
x0 = [2, 1] and use HZ
x0 = [2, 2] and use HZ
Initial condition: y = 1.5254, x = [0 3]Iteration: 1 	y = 0.0004, x = [ 0.0000 -0.0389], gradient = 0.0195Iteration: 2 	y = 0.0000, x = [ 0.0000 -0.0012], gradient = 0.0006Iteration: 3 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Iteration: 4 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [ 0.0000  0.0000]
Initial condition: y = 1.3176, x = [2 1]Iteration: 1 	y = 0.5170, x = [ 0.2435  1.4838], gradient = 0.6077Iteration: 2 	y = 0.3029, x = [-0.3727  1.0246], gradient = 0.5152Iteration: 3 	y = 0.1579, x = [-0.5426  0.2577], gradient = 0.5198Iteration: 4 	y = 0.0758, x = [-0.3477 -0.2603], gradient = 0.3566Iteration: 5 	y = 0.0462, x = [-0.1203 -0.3975], gradient = 0.2265Iteration: 6 	y = 0.0329, x = [ 0.0433 -0.3585], gradient = 0.1822Iteration: 7 	y = 0.0220, x = [ 0.1482 -0.2118], gradient = 0.1796Iteration: 8 	y = 0.0120, x = [ 0.1548 -0.0102], gradient = 0.1543Iteration: 9 	y = 0.0064, x = [ 0.0865  0.1037], gradient = 0.1005Iteration: 10 	y = 0.0042, x = [ 0.0221  0.1262], gradient = 0.0668Iteration: 11 	y = 0.0031, x = [-0.0241  0.1051], gradient = 0.0578Iteration: 12 	y = 0.0020, x = [-0.0511  0.0513], gradient = 0.0571Iteration: 13 	y = 0.0010, x = [-0.0451 -0.0090], gradient = 0.0453Iteration: 14 	y = 0.0006, x = [-0.0222 -0.0367], gradient = 0.0288Iteration: 15 	y = 0.0004, x = [-0.0031 -0.0397], gradient = 0.0201Iteration: 16 	y = 0.0003, x = [ 0.0104 -0.0304], gradient = 0.0184Iteration: 17 	y = 0.0002, x = [ 0.0169 -0.0114], gradient = 0.0178Iteration: 18 	y = 0.0001, x = [ 0.0128  0.0059], gradient = 0.0132Iteration: 19 	y = 0.0001, x = [ 0.0055  0.0124], gradient = 0.0083Iteration: 20 	y = 0.0000, x = [-0.0001  0.0123], gradient = 0.0062Iteration: 21 	y = 0.0000, x = [-0.0041  0.0085], gradient = 0.0059Iteration: 22 	y = 0.0000, x = [-0.0053  0.0021], gradient = 0.0055Iteration: 23 	y = 0.0000, x = [-0.0035 -0.0027], gradient = 0.0038Iteration: 24 	y = 0.0000, x = [-0.0013 -0.0041], gradient = 0.0024Iteration: 25 	y = 0.0000, x = [ 0.0004 -0.0037], gradient = 0.0019Iteration: 26 	y = 0.0000, x = [ 0.0015 -0.0023], gradient = 0.0019Iteration: 27 	y = 0.0000, x = [ 0.0016 -0.0002], gradient = 0.0016Iteration: 28 	y = 0.0000, x = [ 0.0010  0.0010], gradient = 0.0011Iteration: 29 	y = 0.0000, x = [ 0.0003  0.0013], gradient = 0.0007Iteration: 30 	y = 0.0000, x = [-0.0002  0.0011], gradient = 0.0006Iteration: 31 	y = 0.0000, x = [-0.0005  0.0006], gradient = 0.0006Iteration: 32 	y = 0.0000, x = [-0.0005 -0.0001], gradient = 0.0005Iteration: 33 	y = 0.0000, x = [-0.0002 -0.0004], gradient = 0.0003Iteration: 34 	y = 0.0000, x = [-0.0000 -0.0004], gradient = 0.0002Iteration: 35 	y = 0.0000, x = [ 0.0001 -0.0003], gradient = 0.0002Iteration: 36 	y = 0.0000, x = [ 0.0002 -0.0001], gradient = 0.0002Iteration: 37 	y = 0.0000, x = [ 0.0001  0.0001], gradient = 0.0001Iteration: 38 	y = 0.0000, x = [ 0.0001  0.0001], gradient = 0.0001Iteration: 39 	y = 0.0000, x = [ 0.0000  0.0001], gradient = 0.0001Iteration: 40 	y = 0.0000, x = [-0.0000  0.0001], gradient = 0.0001Iteration: 41 	y = 0.0000, x = [-0.0001  0.0000], gradient = 0.0001Iteration: 42 	y = 0.0000, x = [-0.0000 -0.0000], gradient = 0.0000Iteration: 43 	y = 0.0000, x = [-0.0000 -0.0000], gradient = 0.0000Iteration: 44 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 45 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 46 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 47 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Iteration: 48 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [ 0.0000  0.0000]
Initial condition: y = 1.3176, x = [2 1]Iteration: 1 	y = 0.5170, x = [ 0.2435  1.4838], gradient = 0.6077Iteration: 2 	y = 0.3194, x = [-0.4134  1.0358], gradient = 0.5266Iteration: 3 	y = 0.0004, x = [-0.0111  0.0355], gradient = 0.0209Iteration: 4 	y = 0.0004, x = [-0.0072  0.0370], gradient = 0.0198Iteration: 5 	y = 0.0000, x = [ 0.0054  0.0041], gradient = 0.0057Iteration: 6 	y = 0.0000, x = [-0.0000  0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [-0.0000  0.0000]
Initial condition: y = 1.3176, x = [2 1]Iteration: 1 	y = 0.5170, x = [ 0.2435  1.4838], gradient = 0.6077Iteration: 2 	y = 0.3072, x = [-0.3836  1.0276], gradient = 0.5182Iteration: 3 	y = 0.0004, x = [ 0.0119  0.0358], gradient = 0.0215Iteration: 4 	y = 0.0002, x = [-0.0077  0.0221], gradient = 0.0135Iteration: 5 	y = 0.0000, x = [-0.0000 -0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [-0.0000 -0.0000]
Initial condition: y = 1.3176, x = [2 1]Iteration: 1 	y = 0.5170, x = [ 0.2435  1.4838], gradient = 0.6077Iteration: 2 	y = 0.2887, x = [-0.4140  0.9624], gradient = 0.5140Iteration: 3 	y = 0.1582, x = [-0.5550  0.1947], gradient = 0.5287Iteration: 4 	y = 0.0763, x = [-0.3343 -0.2981], gradient = 0.3502Iteration: 5 	y = 0.0480, x = [-0.1058 -0.4143], gradient = 0.2270Iteration: 6 	y = 0.0341, x = [ 0.0594 -0.3606], gradient = 0.1872Iteration: 7 	y = 0.0225, x = [ 0.1599 -0.1985], gradient = 0.1855Iteration: 8 	y = 0.0120, x = [ 0.1550  0.0073], gradient = 0.1545Iteration: 9 	y = 0.0065, x = [ 0.0819  0.1131], gradient = 0.0991Iteration: 10 	y = 0.0044, x = [ 0.0172  0.1302], gradient = 0.0673Iteration: 11 	y = 0.0032, x = [-0.0290  0.1046], gradient = 0.0598Iteration: 12 	y = 0.0020, x = [-0.0540  0.0461], gradient = 0.0587Iteration: 13 	y = 0.0010, x = [-0.0445 -0.0140], gradient = 0.0451Iteration: 14 	y = 0.0006, x = [-0.0206 -0.0392], gradient = 0.0284Iteration: 15 	y = 0.0004, x = [-0.0016 -0.0407], gradient = 0.0204Iteration: 16 	y = 0.0003, x = [ 0.0119 -0.0299], gradient = 0.0191Iteration: 17 	y = 0.0002, x = [ 0.0175 -0.0096], gradient = 0.0182Iteration: 18 	y = 0.0001, x = [ 0.0125  0.0074], gradient = 0.0130Iteration: 19 	y = 0.0001, x = [ 0.0050  0.0131], gradient = 0.0082Iteration: 20 	y = 0.0000, x = [-0.0006  0.0125], gradient = 0.0063Iteration: 21 	y = 0.0000, x = [-0.0045  0.0082], gradient = 0.0061Iteration: 22 	y = 0.0000, x = [-0.0055  0.0015], gradient = 0.0055Iteration: 23 	y = 0.0000, x = [-0.0034 -0.0031], gradient = 0.0037Iteration: 24 	y = 0.0000, x = [-0.0011 -0.0043], gradient = 0.0024Iteration: 25 	y = 0.0000, x = [ 0.0005 -0.0038], gradient = 0.0020Iteration: 26 	y = 0.0000, x = [ 0.0016 -0.0022], gradient = 0.0020Iteration: 27 	y = 0.0000, x = [ 0.0016 -0.0000], gradient = 0.0016Iteration: 28 	y = 0.0000, x = [ 0.0009  0.0011], gradient = 0.0011Iteration: 29 	y = 0.0000, x = [ 0.0002  0.0014], gradient = 0.0007Iteration: 30 	y = 0.0000, x = [-0.0003  0.0011], gradient = 0.0006Iteration: 31 	y = 0.0000, x = [-0.0006  0.0005], gradient = 0.0006Iteration: 32 	y = 0.0000, x = [-0.0005 -0.0001], gradient = 0.0005Iteration: 33 	y = 0.0000, x = [-0.0002 -0.0004], gradient = 0.0003Iteration: 34 	y = 0.0000, x = [-0.0000 -0.0004], gradient = 0.0002Iteration: 35 	y = 0.0000, x = [ 0.0001 -0.0003], gradient = 0.0002Iteration: 36 	y = 0.0000, x = [ 0.0002 -0.0001], gradient = 0.0002Iteration: 37 	y = 0.0000, x = [ 0.0001  0.0001], gradient = 0.0001Iteration: 38 	y = 0.0000, x = [ 0.0001  0.0001], gradient = 0.0001Iteration: 39 	y = 0.0000, x = [-0.0000  0.0001], gradient = 0.0001Iteration: 40 	y = 0.0000, x = [-0.0000  0.0001], gradient = 0.0001Iteration: 41 	y = 0.0000, x = [-0.0001  0.0000], gradient = 0.0001Iteration: 42 	y = 0.0000, x = [-0.0000 -0.0000], gradient = 0.0000Iteration: 43 	y = 0.0000, x = [-0.0000 -0.0000], gradient = 0.0000Iteration: 44 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 45 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 46 	y = 0.0000, x = [ 0.0000 -0.0000], gradient = 0.0000Iteration: 47 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Iteration: 48 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [ 0.0000  0.0000]
Initial condition: y = 1.3176, x = [2 1]Iteration: 1 	y = 0.5170, x = [ 0.2435  1.4838], gradient = 0.6077Iteration: 2 	y = 0.2369, x = [-0.4269  0.8147], gradient = 0.4939Iteration: 3 	y = 0.0000, x = [-0.0064 -0.0053], gradient = 0.0070Iteration: 4 	y = 0.0000, x = [ 0.0003 -0.0008], gradient = 0.0005Iteration: 5 	y = 0.0000, x = [ 0.0000  0.0000], gradient = 0.0000Solution: 	y = 0.0000, x = [ 0.0000  0.0000]
Initial condition: y = 1.0669, x = [2 2]Iteration: 1 	y = 0.9666, x = [ 1.7144  2.5793], gradient = 0.2652Iteration: 2 	y = 0.0075, x = [ 3.1474  4.4532], gradient = 0.0104Iteration: 3 	y = 0.0074, x = [ 3.1379  4.4420], gradient = 0.0028Iteration: 4 	y = 0.0074, x = [ 3.1400  4.4384], gradient = 0.0000Solution: 	y = 0.0074, x = [ 3.1400  4.4384]
From these examples, we see that adopting the Conjugate Gradient algorithm for general function optimization will no longer ensure the process is terminated at some particular number of iteration. That’s why we safeguard the algorithm using max_iter.
The difference in the βk calculation method results in a different performance of the algorithm. Also, a poor initialization of xk may not lead to the global minimum in the process.
What else could you infer from these results? Let’s discuss this in the response section below.
Conjugate Gradient algorithm is used to solve a linear system, or equivalently, optimize a quadratic convex function. It sets the learning path direction such that they are conjugates with respect to the coefficient matrix A and hence the process is terminated after at most the dimension of A iterations. The algorithm can be adopted for optimizing general differentiable functions leveraging Wolfe Line Search and can be improved by considering variations of direction coefficient βk.
You might want to continue reading my related stories:
Albers Uzila
Thanks! If you enjoy this story and want to support me as a writer, consider becoming a member. For only $5 a month, you’ll get unlimited access to all stories on Medium. If you sign up using my link, I’ll earn a small commission.
If you’re one of my referred Medium members, feel free to email me at geoclid.members[at]gmail.com to get the complete python code of this story.