Algebra with Numpy and Scipy | by Andrea Castiglioni  | Towards Data Science

In this tutorial I want to revise some basics concepts of linear algebra, least square minimization and curve fitting which are useful tools for any scientist working his way trough data analysis in python. These tools can be applied to a big variety of problems, from linear regression to ODE (ordinary differential equation).
We will start with the basics working our way to more complicated cases using the tools provided from numpy and scipy (built on top of numpy): two popular scientific computing packages for python. In this article I am using those versions:numpy ‘1.20.0’scipy ‘1.6.0’but the routines we will use are not new ones, so there should be no problems in using an older version.
By the end of this article you will be able to implement your code for solving common algebra calculation on matrices and arrays, fit your data with a custom equation and apply least square regression to your problems.
I employed these techniques in some other articles like Multiple linear regression, Linear Regression, and some advanced least square minimization — curve fit with a basic analysis of covid cases with SIR model.
From high school or college we all had our chance to deal with systems of linear equations. They looked pretty or nasty but was basically something like:
The task in this problems is to find the x and y that satisfy the relationship. We can solve this manually by writing x = 1-y from the second equation and substitute it in the first equation that becomes: (1-y) + (2y) = 0. The solution is y = -1 and x = 2.
As soon as we progressed in our studies, these equations eventually became matrices. The above equations can be written as:
Where A is a 2x2 matrix and its called the coefficient matrix.and b is a colum vector, or a 2x1 matrix and represent the ordinate or “dependent variable” values. x is the vector (or matrix) we have to solve this system for. Notice that in this representation all the terms like x,y,t,... are condensed in the x.
From matrix multiplication rules we know that if A has m rows and n columns (mxn) and x has n rows and p columns (nxp), then its product Ax will have m rows and p columns (mxp). To be a linear system, A is a square matrix, it will be mxm, so x has to be mxp and b has the same dimensions mxp.
In python this becomes:
The output will be like:
Now we can solve our system simply with np.linalg.solve:
and x will be:
which is exactly the solution we found by hand! Of course we can experiment a little here. We will find that with increasing complexity of our systems we are able to solve them easily this way. We can check that our solution matches our b vector by taking the dot product of A and x using np.dot(a,x). According to np.dot documentation:
If both a and b are 1-D arrays, it is inner product of vectors (without complex conjugation).
If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a @ b is preferred.
In this case A is a matrix and x is a vector, we can test that all three methods provide the same results:
# different way to apply inner product or matrix multiplication>> np.dot(a,x), a@x, np.matmul(a,x)array([0., 1.]), array([0., 1.]), array([0., 1.]))
a = np.array([[1,2,1,1],[1,1,3,4],[3,4,1,3],[5,6,1,0]])b = np.array([0,1,2,1])x = np.linalg.solve(a, b)xarray([ 0.8, -0.4, -0.6,  0.6])
Unfortunately, if we play around a bit we will soon find out that this approach only works for square matrices. But real life data usually is not in that format. In the next session we will see how to cope with that.
We saw that linalg.solve(a,b) can give us the solution of our system. Sometimes, however, it is not possible to find an exact solution and we are happy with the best approximate solution. Here comes least square.
numpy.linalg.lstsq Return the least-squares solution to a linear matrix equation.
Why bother? Well when we solve a system algebrically like before, we need the A matrix to be a square matrix. Sometimes we are not that lucky! So what linalg.solve does is to computes the vector x that approximatively solves the equation a @ x = b. The equation may be under-, well-, or over-determined (i.e., the number of linearly independent rows of a can be less than, equal to, or greater than its number of linearly independent columns). If a is square a then x (but for round-off error) is the “exact” solution of the equation. Else, x minimizes the Euclidean 2-norm ||b-ax||.
Let’s see the first example with the last system we had:
x,res,rank,s = np.linalg.lstsq(a, b, rcond=None)xarray([ 0.8, -0.4, -0.6,  0.6])resarray([], dtype=float64rank4sarray([3.35227247, 0.96338284, 0.47702795, 0.26750408, 0.07920752]))
yay! The output is the same as np.linalg.solve(). The other values are:res: sums of squared residuals, rank: the rank of the matrix a, s: singular values of a.
If we try to solve a system like this: we create a matrix composed of random numbers with shape 6 rows and 5 columns and we create the be vector by selecting a random number from 1 to 4 with shape 6.
A = np.random.rand(6,5)bb = np.random.choice([1,2,3,4],6)np.linalg.solve(A, bb)
We get this error: the matrix is not square.
However, we can use leastsqr:
x,res,rank,s = np.linalg.lstsq(A, bb, rcond=None)x = array([ 2.386104  , -2.1627241 ,  0.04946452,  2.01325397,  1.04230874]),res =  array([2.9453122]),rank =  5,s = array([3.21281719, 0.99382227, 0.56108033, 0.27539112, 0.02256652])
Here we saw that the resitual is now not 0, but we solved our system! The results may vary due to the different random number generation of the previous step.
How can we use such a tool?Well, for example to fit a line: let’s imagine we have a dependent variable x which might be the time of some process, and y is the response of our system. We’re interested in finding the best fit of such a relationship:
#Fit a line, y = mx + c, x = np.array([0, 1, 2, 3, 4, 5, 6])y = np.array([-1, 0.2, 0.9, 2.1, 3, 2.8, 4.2])
our matrix A is now the x array, to which we add a column of
# we create an array with the same shape of x, filled with 1ones = np.ones(len(x))# we then concatenate the two arrays, and transpose themA = np.vstack([x, ones]).TA
By using the previous lines of code:
q,res,rank,s=np.linalg.lstsq(A, y, rcond=None)q = array([ 0.81785714, -0.71071429]),res = array([0.74821429])rank = 2s = array([9.79578621, 1.42918595])
our solution is the q line, which we can express with the same nomenclature as before: #y = mx + c,
# notice we are taking the first (0) argument from the functionm, c = np.linalg.lstsq(A, y, rcond=None)[0]m = 0.81785714c = -0.71071429
So the slope is positive while the intercept is negative. It is always good practice to actually plot your data.
import matplotlib.pyplot as plt_ = plt.plot(x, y, 'o', label='Original data', markersize=10)_ = plt.plot(x, m*x + c, 'r', label='Fitted line')_ = plt.legend()plt.show()
We can explore also the residuals easily:
# remember to use parenthesis to correctly calculate residuals_ = plt.plot(y, y-(m*x + c), 'ro',label='Residuals', markersize=10)_ = plt.legend()plt.axhline(0,c='k')plt.show()
And if we take the square sum of them, we end up with the residual results from linalg.lstsq.
np.sum((y-(m*x + c))**2)0.7482142857142864
Sometimes we are interested in relationships which are not linear, in such case we wonder how can we approximate our data. Let’s generate 200 datapoints from 0 to 20 with numpy linspace function. Then we create a sinusoidal signal with an offset, a multiplication factor and some noise.
n = 200t = np.linspace(0,20,n)y = 3.0 + 2.0 * np.sin(t) + np.random.randn(n)plt.plot(t, y, 'o')
We apply the same procedure we saw above with linalg.lstsq:
# Notice the use of np.c_ which translates slice objects# to concatenation along the second axis# equivalent to np.vstack([x, ones]).Ta = np.c_[np.ones(n),np.sin(t)]x,res,rank,s = np.linalg.lstsq(a, y, rcond=None)
Note that we know that our function is sinusoidal, we are not random guessing the functional form of this data. The solution found is:x = array([3.05179667, 1.97737329]). If we compare its values to the ones we defined by hand we see that it is pretty close. y = 3.0 + 2.0 * np.sin(t) + np.random.randn(n)
We can check how good the solution is by plotting:
plt.plot(t, y, 'o')plt.plot(t, np.dot(a,x))
Residuals are around 200 in my case. We can calculate R2 from sklearn.metrics:
from sklearn.metrics import r2_scorer2_score(y,np.dot(a,x))>> 0.658482073384907
0.65 is not bad. However, let’s say we know that our function is periodic. Only we don’t know if it’s sin or cos. If we ran our code with the latter, the result isn’t as satisfying:
a = np.c_[np.ones(n),np.cos(t)]x,res,rank,s = np.linalg.lstsq(a, y, rcond=None)plt.plot(t, y, 'o')plt.plot(t, np.dot(a,x))
Now the result is not as satisfying: residuals jumped to 600 and R2 is basically 0.
res, r2_score(y,np.dot(a,x))(array([615.9314504]), 0.001246313173887681)
In this situation we can make use of handy function from scipy.optimize called curve_fit. All we have to do is import the package, define the function of which we want to optimize the parameters, and let the package do the magic. Curve_fit use non-linear least squares to fit a function, f, to data. Assumes ydata = f(xdata, *params) + eps. where eps is the residual.
So below we define our function with 3 parameters: the offset a, the multiplicative factor b and an extra argument omega for the sinusoidal function. The standard code is like:
popt will store the optimal values for the parameters so that the sum of the squared residuals of f(xdata, *popt) - ydata is minimized. pcov stores the estimated covariance of popt. The diagonals provide the variance of the parameter estimate. To compute one standard deviation errors on the parameters use perr = np.sqrt(np.diag(pcov)).:
array([2.98659897, 2.03908412, 0.9977973 ])plt.plot(t, y, 'o')plt.plot(t, funfit(t, *popt), 'r-',         label='fit: a=%5.3f, b=%5.3f, omega=%5.3f' % tuple(popt))plt.legend()plt.show()
So a is close to 3 and b is close to 2 and the multiplication factor omega is close to 1. We can check our pcov matrix and calculate the error in the optimal values:
perr = np.sqrt(np.diag(pcov))array([0.06914534, 0.09840551, 0.00423498])
So we see that our error is around 2% for a, 5% for b and 0.4% for omega. However, little they tell us about how good the fit is to our data. R2 is 0.664 which is close to the previously calculated with np.linalg.lstsq. What if we tried with cosine?
def fun2fit(t, a, b, omega):    return a + b*np.cos(omega*t)popt, pcov = curve_fit(fun2fit, t, y)plt.plot(t, y, 'o')plt.plot(t, fun2fit(t, *popt), 'r-',         label='fit: a=%5.3f, omega=%5.3f, q=%5.3f' % tuple(popt))plt.legend()plt.show()
perr = array([0.09319211, 0.13281591, 0.00744385])
Errors are now around 3% for a, 8% for b and 0.7% for omega. R2 = 0.387 in this case.The fit is now better than our previous attempt with the use of simple leastsq. But it could be better. We can generalize our equation by adding another parameter, an offset to the argument of the cosine.
def fun3fit(t, a, b, omega,q):    return a + b*np.cos(omega*t+q)popt, pcov = curve_fit(fun3fit, t, y)plt.plot(t, y, 'o')plt.plot(t, fun3fit(t, *popt), 'r-',         label='fit: a=%5.3f, b=%5.3f, omega=%5.3f, q=%5.3f' % tuple(popt))plt.legend()plt.show()
The fit is much better. R2 = 0.664.If we ran the exact same code, but exchange cos with sin, we get another optimal parameter for q:
def fun3fit(t, a, b, omega,q):    return a + b*np.sin(omega*t+q)popt, pcov = curve_fit(fun3fit, t, y)plt.plot(t, y, 'o')plt.plot(t, fun3fit(t, *popt), 'r-',         label='fit: a=%5.3f, b=%5.3f, omega=%5.3f, q=%5.3f' % tuple(popt))plt.legend()plt.show()
Again we see that the fitting parameters are close to our equation:y = 3.0 + 2.0 * np.sin(t) where q was 0.
We started this tutorial with some basic concepts about solving algebric systems with numpy linalg.solve. We then saw how to solve systems in which the matrix is not square (we have more equations or less then necessary to solve the exact problem) with numpy linalg.lstsq. We applied this method to a simple linear regression first and then to a nonlinear regression with a periodic function. We saw how to calculate residuals and check the results with the output from linal.lstsq.
Finally, in the case we need more degree of freedom, we saw how we can use scipy optimize.curve_fit. Here we can build our function and evaluate optimal parameters and residuals as well.
Whole code is available on github.