Modeling Functions. From linear to logistic regression | by Shinichi Okada | Towards Data Science

Table of ContentsIntroduction1. Linear models2. Quadratic models3. Cubic models4. Exponential models5. Logarithmic models6. Sinusoidal models7. Logistic modelsConclusion
We are going to draw a scatter graph and model a regression line from linear to logistic with Jupyter Notebook.
The first one is a linear model. A linear model is express as y=mx+c. We are going to use numpy.array or numpy.arange to create data. If you want to read more about a linear relationship, please read A Measure of Linear Relationship. We import Python libraries numpy and matplotlib. We create a year and a co2 array.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineyear=np.array([1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000])co2=np.array([338.7,341.1,344.4,347.2,351.5,354.2,356.4,358.9,362.6,366.6,369.4])
First, we create a scatter plot using matplotlib. Add the title, label, x, and y-axis labels. You need to use show() method. You can plot without it but this will remove unnecessary outputs.
plt.scatter(year,co2,label='CO2')plt.title("Year vs CO2")plt.xlabel('Year')plt.ylabel('CO2')plt.legend()plt.show()
As you can see in the above graph, you have decimals in the x-axis. We use the first three lines to make them integers in the following codes.
from matplotlib.ticker import MaxNLocatorax = plt.figure().gca()ax.xaxis.set_major_locator(MaxNLocator(integer=True))plt.scatter(year,co2,label='CO2')plt.title("Year vs CO2")plt.xlabel('Year')plt.ylabel('CO2')plt.legend()plt.show()
The easiest way is to use numpy.polyfit. By setting order to 1, it will return an array of linear coefficients. Using it in numpy.poly1d returns an equation using the coefficients.
import numpy as npimport matplotlib.pyplot as pltfrom matplotlib.ticker import MaxNLocatorfrom sklearn.linear_model import LinearRegression%matplotlib inlineax = plt.figure().gca()ax.xaxis.set_major_locator(MaxNLocator(integer=True))year=np.array([1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000])co2=np.array([338.7,341.1,344.4,347.2,351.5,354.2,356.4,358.9,362.6,366.6,369.4])coef = np.polyfit(year, co2, 1)equ = np.poly1d(coef)x_plot = np.linspace(1975,2005,100)y_plot = equ(x_plot)plt.plot(x_plot, y_plot, color='r')plt.scatter(year,co2,label='CO2')plt.title("Year vs CO2")plt.xlabel('Year')plt.ylabel('CO2')plt.legend()plt.show()
The second way to find the regression slope and intercept is to use sklearn.linear_model.LinearRegression. This class requires the x values to be one column. We modify year data using reshape(-1,1). The original year data has 1 by 11 shape. You need to reshape the year data to 11 by 1.
year1=year.reshape((-1,1))print(np.shape(year))print(np.shape(year1))
We import sklearn.linear_model.LinearRegression, reshape the year data, fit our data using LinearRegression().fit(). This will return the slope, coef_ and the y-intercept, intercept_. coef_ returns an array, so we take the first item by using reg.coef_[0]. Let’s print out our regression line equation.
from sklearn.linear_model import LinearRegressionyear1=year.reshape((-1,1))reg = LinearRegression().fit(year1,co2)slope=reg.coef_[0]intercept=reg.intercept_print(f'The equation of regression line is y={slope:.3f}x+{intercept:.3f}.')
We draw a scatter plot and our linear regression line together. We use a new x domain from 1975 to 2005 taking 100 samples for the regression line, np.linspace(1975,2005,100). Then using x domain, slope, and y-intercept to draw a regression line.
import numpy as npimport matplotlib.pyplot as pltfrom matplotlib.ticker import MaxNLocatorfrom sklearn.linear_model import LinearRegression%matplotlib inlineax = plt.figure().gca()ax.xaxis.set_major_locator(MaxNLocator(integer=True))year=np.array([1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000])co2=np.array([338.7,341.1,344.4,347.2,351.5,354.2,356.4,358.9,362.6,366.6,369.4])year1=year.reshape((-1,1))reg = LinearRegression().fit(year1,co2) slope=reg.coef_[0]intercept=reg.intercept_plt.scatter(year,co2,label='CO2')X_plot = np.linspace(1975,2005,100)Y_plot = slope*X_plot+interceptplt.plot(X_plot, Y_plot, color='r')plt.title("Year vs CO2")plt.xlabel('Year')plt.ylabel('CO2')plt.legend()plt.show()print(f'The equation of regression line is y={slope:.3f}x+{intercept:.3f}.')
Another way to find the regression slope and intercept is to use scipy.stats.linregress. This returns slope, intercept, rvalue, pvalue, stderr.
from scipy.stats import linregressslope, intercept, r_value, p_value, std_err = linregress(year,co2)print(f'The equation of regression line is y={slope:.3f}x+{intercept:.3f}.')
To draw a line we need x points. We use np.linspace which is numpy.linspace since we used import numpy as np. Our data is from 1975 to 2000. So let's use 1960 for start, 2005 for stop and 100 for the number of samples.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.stats import linregress%matplotlib inlineyear=np.array([1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000])co2=np.array([338.7,341.1,344.4,347.2,351.5,354.2,356.4,358.9,362.6,366.6,369.4])X_plot = np.linspace(1975,2005,100)Y_plot = slope*X_plot+interceptplt.plot(X_plot, Y_plot, color='r')plt.show()
Now we put a scatter plot, regression line, and a regression equation together.
import numpy as npimport matplotlib.pyplot as pltfrom matplotlib.ticker import MaxNLocatorfrom scipy.stats import linregress%matplotlib inlineax = plt.figure().gca()ax.xaxis.set_major_locator(MaxNLocator(integer=True))slope, intercept, r_value, p_value, std_err = linregress(year,co2)X_plot = np.linspace(1975,2005,100)Y_plot = slope*X_plot+interceptplt.plot(X_plot, Y_plot, color='r')plt.scatter(year,co2,label='CO2')plt.title("Year vs CO2")plt.xlabel('Year')plt.ylabel('CO2')plt.legend()plt.show()print(f'The equation of regression line is y={slope:.3f}x+{intercept:.3f}.')
Use the following data to graph a scatter plot and regression line. Find a linear regression equation.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetemp = np.array([55,60,65,70,75,80,85,90])rate = np.array([45,80,92,114,141,174,202,226])
Did you draw a scatter and regression graph? The regression line should be y=5.119x−236.88.
towardsdatascience.com
We use Numpy’s arange to create 10 integers from 0 to 9.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetime = np.arange(10)height = np.array([450,445,430,409,375,331,280,215,144,59])
Let’s plot the above data.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineplt.scatter(time,height,label='Height of a ball')plt.title("Time vs Height")plt.xlabel('Time')plt.ylabel('Height')plt.legend()plt.show()
numpy.polyfit fits a polynomial. It requires x, y, and degree of the fitting polynomial. 2 for a quadratic, 3 for a cubic, etc. It returns an array with polynomial coefficients from a higher power to the constant. For a quadratic function, these are a, b and c in:
coef = np.polyfit(time, height, 2)coef
Let’s print out the quadratic regression line.
print(f'The equation of regression line is y=')print(equ)
Or using coefficients the regression line is:
print(f'The equation of regression line is y={coef[0]:.3f}x^2+{coef[1]:.3f}x+{coef[2]:.3f}.')
We use NumPy's poly1d and polyfitagain. np.poly1d(coefficients) will return a polynomial equation using our coefficient.
equ = np.poly1d(coef)
We can find a value for any x. For example, if you want to find y value when x=1:
equ(1)
We use this to draw our regression line. We use numpy.linspace to define x values from 0 to 10 for 100 samples. And use it in the equ for y values.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinex_plot = np.linspace(0,10,100)y_plot = equ(x_plot)plt.plot(x_plot, y_plot, color='r')plt.show()
We put it all together.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetime = np.arange(10)height = np.array([450,445,430,409,375,331,280,215,144,59])coef = np.polyfit(time, height, 2)equ = np.poly1d(coef)x_plot = np.linspace(0,10,100)y_plot = equ(x_plot)plt.plot(x_plot, y_plot, color='r')plt.scatter(time,height,label='Height of a ball')plt.title("Time vs Height")plt.xlabel('Time')plt.ylabel('Height')plt.legend()plt.show()print(f'The equation of regression line is y=')print(equ)
By using the following data, draw a scatter plot and regression line in a graph. Find the quadratic regression equation.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineangle = np.arange(20,80,10)distance = np.array([371,465,511,498,439,325])
Could you draw a scatter and regression line? The quadratic equation should be:
towardsdatascience.com
You can use the same method as the quadratic function above. We are going to use plyfit and poly1d. First, we prepare the data. And let’s draw a scatter plot.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineengspeed = np.arange(9,23,2)avespeed = np.array([6.45,7.44,8.88,9.66,10.98,12.56,15.44])plt.scatter(engspeed,avespeed,label='Speed of different boat engine')plt.title("Average speed of different boat engine")plt.xlabel('Engine speed')plt.ylabel('Boad speed')plt.ylim(0,20)plt.legend()plt.show()
Using polyfit returns coefficients. For a cubic function, a, b, c, and d in:
coef = np.polyfit(engspeed, avespeed, 3)print(coef)
We put all together.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineengspeed = np.arange(9,23,2)avespeed = np.array([6.45,7.44,8.88,9.66,10.98,12.56,15.44])plt.scatter(engspeed,avespeed,label='Speed of different boat engine')coef = np.polyfit(engspeed, avespeed, 3)equ = np.poly1d(coef)x_plot = np.linspace(8,25,100)y_plot = equ(x_plot)plt.plot(x_plot, y_plot, color='r')plt.title("Average speed of different boat engine")plt.xlabel('Engine speed')plt.ylabel('Boad speed')plt.ylim(0,20)plt.legend()plt.show()a, b, c, d = coefprint(f'The equation of regression line is y={a:.3f}x^3+{b:.3f}x^2+{c:.3f}x+{d}.')
Draw a scatter plot and a cubic regression line using the following data. Print the cubic equation.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinex=np.arange(1,8)y=np.array([0,0.012,0.06,0.162,0.336,0.6,0.972])
Could you plot a scatter and a regression line? The regression line equation should be:
The coefficients are [ 3.00000000e-03, -1.16796094e-16, -9.00000000e-03, 6.00000000e-03]. These means:
The second one is practically 0. Try the following to see both are the same 0.3.
print(300e-03)print(300*10**(-3))
We are going to explore three exponential models.
Let’s set up the data. And draw a scatter plot.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineday = np.arange(0,8)weight = np.array([251,209,157,129,103,81,66,49])plt.scatter(day,weight,label='Weight change')plt.title("Day vs Weight")plt.xlabel('Day')plt.ylabel('Weight')plt.legend()plt.show()
We are going to use scipy.optimize.curve_fit. This requires a function, x and y values, and initial values, p0 in array form. Finding proper p0 is a bit of trial and error. You have to test different values. We use p0=(1, 1e-6, 1). It returns optimal values for the parameters and the estimated covariance of popt.
Our first exponential function uses a, b, and c. We are going to define a function first. This is used in the curve_fit method. For a smooth curve, we set x values using numpy.linspace from 0 to 7 with 100 samples.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlinedef func(x, a, b, c):    return a * np.exp(-b * x) + cpopt, pcov = curve_fit(func, day, weight, p0=[1, 1e-6, 1])x_plot=np.linspace(0,7,100)plt.plot(x_plot, func(x_plot, *popt), 'r-')plt.scatter(day,weight,label='Day vs Weight')plt.title("Day vs Weight a*e^-bx +c")plt.xlabel('Day')plt.ylabel('Weight')plt.legend()plt.show()# equationa=popt[0].round(2)b=popt[1].round(2)c=popt[2].round(2)print(f'The equation of regression line is y={a}e^({b}x)+{c}')
The second function uses a and b. We define the function accordingly.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlinedef func2(x, a, b):    return a * np.exp(-b * x)popt, pcov = curve_fit(func2, day, weight, p0=[1, 1e-6])x_plot=np.linspace(0,7,100)plt.plot(x_plot, func2(x_plot, *popt), 'r-')plt.scatter(day,weight,label='Day vs Weight')plt.title("Day vs Weight a*e^-bx")plt.xlabel('Day')plt.ylabel('Weight')plt.legend()plt.show()# equationa=popt[0].round(2)b=popt[1].round(2)print(f'The equation of regression line is y={a}e^({b}x)')
The last exponential function uses a and b. We modify the function accordingly.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlinedef func3(x, a, b):    return a * b ** xpopt, pcov = curve_fit(func3, day, weight, p0=[1, 1e-6])x_plot=np.linspace(0,7,100)plt.plot(x_plot, func3(x_plot, *popt), 'r-')plt.scatter(day,weight,label='Day vs Weight')plt.title("Day vs Weight a*b^x")plt.xlabel('Day')plt.ylabel('Weight')plt.legend()plt.show()# equationa=popt[0].round(4)b=popt[1].round(4)print(f'The equation of regression line is y={a}*{b}^x')
TI Nspire’s exponential regression fits the model equation y=ab^x to the data using a least-squares fit on transformed values x and ln(y). It returns a different value.
Using the following data find an exponential function in the form of ab^x. Plot a scatter diagram and draw a regression line.
import numpy as npweek = np.arange(1,21)views = np.array([102365, 38716,21617,24305,9321,14148,2103,8285,5098,3777,831,1007,834,34,378,204,6,42,54,31])
Generally, we use the semi-log model for an exponential function:
We set up our mock data and plot a scatter diagram.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetime = np.arange(0,30,4)bacteria = np.array([20,150,453,920,1820,9765,15487,19450])plt.scatter(time,bacteria,label='Bacteria')plt.title("Time vs Bacteria")plt.xlabel('time')plt.ylabel('bacteria')plt.legend()plt.show()
We are going to take a natural log for the bacteria values by using numpy.log. numpy.log is the natural logarithm. This should show a linear trend. We need to modify the title and y label with ln(bacteria).
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetime = np.arange(0,30,4)bacteria = np.array([20,150,453,920,1820,9765,15487,19450])plt.scatter(time,np.log(bacteria),label='Bacteria')plt.title("Time vs ln(Bacteria)")plt.xlabel('time')plt.ylabel('ln(bacteria)')plt.legend()plt.show()
We use numpy.polyfit which we used in a quadratic and cubic function. We use 1 in numpy.polyfit() so that it will return a linear regression. numpy.polyfit returns all coefficients for our equation.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinetime = np.arange(0,30,4)bacteria = np.array([20,150,453,920,1820,9765,15487,19450])p = np.polyfit(time, np.log(bacteria), 1)plt.plot(time, p[0] * time + p[1], 'g--', label='Semi-log graph')plt.scatter(time,np.log(bacteria),label='Bacteria')plt.title("Time vs Bacteria")plt.xlabel('time')plt.ylabel('bacteria')plt.legend()plt.show()print(f'The equation of regression line is y={p[0]:.3f} * x + {p[1]:.3f}')
The log-log model is used for a power function.
Let’s set up data and plot a scatter diagram.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinex=np.array([2,30,70,100,150])y=np.array([4.24,16.4,25.1,30,36.7])plt.scatter(x,y,label='Log-log')plt.title("Log-Log model")plt.xlabel('x')plt.ylabel('y')plt.legend()plt.show()
We take a natural log for x and y values using numpy.log. We need to modify the x and y labels to ln(x) and ln(y).
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlinex=np.array([2,30,70,100,150])y=np.array([4.24,16.4,25.1,30,36.7])p = np.polyfit(np.log(x), np.log(y), 1)plt.plot(np.log(x), p[0] * np.log(x) + p[1], 'r--', label='Regression line')plt.scatter(np.log(x),np.log(y),label='log-log')plt.title("Log-log regression")plt.xlabel('ln(x)')plt.ylabel('ln(y)')plt.legend()plt.show()print(f'The equation of regression line is ln(y)={p[0]:.3f} * ln(x) + {p[1]:.3f}')
Let’s try a sinusoidal function. We set our data and plot a scatter graph. Since we are going to use scipy.optimize.curve_fit, let's import it as well. We used it in the exponential models. We set up our data and draw a scatter plot.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlineyear=np.arange(0,24,2)population=np.array([10.2,11.1,12,11.7,10.6,10,10.6,11.7,12,11.1,10.2,10.2])plt.scatter(year,population,label='Population')plt.title("Year vs Population")plt.xlabel('Year')plt.ylabel('Population')plt.legend()plt.show()
We define a function called sinfunc. This takes parameters x, a, b, c, d. We use numpy.radians for c.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlineyear=np.arange(0,24,2)population=np.array([10.2,11.1,12,11.7,10.6,10,10.6,11.7,12,11.1,10.2,10.2])def sinfunc(x, a, b, c, d):    return a * np.sin(b * (x - np.radians(c)))+dpopt, pcov = curve_fit(sinfunc, year, population, p0=[1,0.4,1,5])x_data = np.linspace(0, 25, num=100)plt.scatter(year,population,label='Population')plt.plot(x_data, sinfunc(x_data, *popt), 'r-',label='Fitted function')plt.title("Year vs Population")plt.xlabel('Year')plt.ylabel('Population')plt.legend()plt.show()a, b, c, d = poptprint(f'The equation of regression line is y={a:.3f} * sin({b:.3f}(x-{np.radians(c):.3f}))+{d:.3f}')
Using the table below, draw a scatter plot and find a cosine regression function.
You may have different coefficients. I used
We set up data and plot a scatter diagram. We set the domain from -10 to 10 and range from 0 to 250, using plt.xlim and plt.ylim. We are going to use scipy.optimize.curve_fit for logistic regression.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlinex=np.arange(0,10)y=np.array([52,133,203,230,237,239.5,239.8,239.9,240,240])plt.scatter(x, y, label='Regression line')plt.title("Logistic regression")plt.xlabel('x')plt.ylabel('y')plt.xlim(-10,10)plt.ylim(0,250)plt.legend()plt.show()
We define our logistic function using logifunc. We find the function parameter in popt using curve_fit. For the regression line, we set a new domain for the function, x_data from -10 to 10. We plot the line using plt.plot.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.optimize import curve_fit%matplotlib inlinex=np.arange(0,10.0)y=np.array([52,133,203,230,237,239.5,239.8,239.9,240,240])def logifunc(x,L,c,k):    return L/ (1 + c*np.exp(-k*x))popt, pcov = curve_fit(logifunc, x, y, p0=[200,1,1])x_data = np.linspace(-10, 10, num=100)plt.scatter(x,y,label='Logistic function')plt.plot(x_data, logifunc(x_data, *popt), 'r-',label='Fitted function')plt.title("Logistic")plt.xlabel('x')plt.ylabel('y')plt.xlim(-10,10)plt.ylim(0,250)plt.legend()plt.show()
Sometimes your data may have negative values in y coordinates.
import pandas as pddf = pd.read_csv('http://bit.ly/2tUIZjK')df.head()
The minimum value for the data needs to be zero. And ideally, the sigmoid midpoint is also zero. But the above data set satisfies neither of them. Using equation (1–2), and adding offset value works for this data set.
x=df.T.iloc[0]y=df.T.iloc[1]def logifunc(x,L,x0,k,off):    return L / (1 + np.exp(-k*(x-x0)))+offplt.scatter(x,y,label='Logistic function')popt, pcov = curve_fit(logifunc, x, y, p0=[50,185,0.1,-222])plt.scatter(x,y,label='Logistic function')x_data = np.linspace(170,205,num=100)plt.plot(x_data, logifunc(x_data, *popt), 'r-',label='Fitted function')plt.legend()plt.show()print(popt)
scipy.optimize.curve_fit was useful for many functions. The only problem was to find good initial values in p0. Sometimes different p0 values return different popt. You can try LMFIT.
Get full access to every story on Medium by becoming a member.