How to predict a variable sinusoid in Python | by Angelica Lo Duca | Towards Data Science

This tutorial describes how to predict a variable sinusoid in Python. Firstly, some sinusoidal data are loaded from a CSV file. Then, data are shaped through a sinusoid curve. After that, linear regression is used to model the minimum and maximum peaks. Finally, the modeled lines are used to calculate the next peaks.
The complete code can be downloaded from my Github repository.
Data analyzed: Arrivals at tourist accommodation establishments extracted from https://ec.europa.eu/eurostat/web/tourism/data/database
Firstly, we read the CSV file and we select only data related to Italy (df[df['geo'] == 'IT']), in the form of numbers (df[df['unit'] == 'NR']) and referred to the sum of all data available (internal and external arrivals) (df[df['c_resid'] == 'TOTAL']).
Then we remove not useful columns through the function drop(). Note that we pass the parameter inplace=True to store effects in the original variable df.
import pandas as pddf = pd.read_csv(‘data/eurostat.csv’)df = df[df[‘geo’] == ‘IT’]df = df[df[‘unit’] == ‘NR’]df = df[df[‘c_resid’] == ‘TOTAL’]df.drop([‘c_resid’, ‘unit’, ‘nace_r2’, ‘2019M11’, ‘geo’],axis=1, inplace=True)df = df.reset_index()df.drop([‘index’],axis=1, inplace=True)
Now we can correct some errors in data: some rows contain the character :, which means no data available. Other rows contain the letter b. We change : in 0 and we remove the letter b.
columns = df.columns[::-1]for column in columns: for row in range(1, len(df[column])): if “b” in df[column][row]: df[column] = df[column][row][:-2] if “:” in df[column][row]: df[column][row] = “0”
Now we calculate the sum over the columns, in order to get the total number of arrivals for each month. We store results in the variable y.
y = []for column in columns: df[column] = df[column].astype(str).astype(int) y.append(df[column].sum())
We plot y to understand the trendline. We build the auxiliary vector X, which contains all the months. Month 0 refers to 1990 Genuary. In order to display x labels in the graph every 5 years, we build two auxiliary variables, x_ticks and x_labels.
import matplotlib.pyplot as pltimport matplotlibimport numpy as npX = np.arange(0, len(columns))step = 5x_ticks = np.arange(0, len(columns)+step, step=12*step)x_labels = []x_labels.append(1990)for i in range(1, len(x_tick)+1): x_labels.append(x_labels[i-1]+step)plt.xticks(x_ticks, x_labels)plt.plot(X, y, color=”red”, linewidth=1)plt.grid()plt.xlabel(‘Months’)plt.ylabel(‘Number of arrivals’)plt.show()
We note that over the last 8 years there was an increase of the maximum peaks, with respect to the previous years. Thus we limit our analysis the last 8 years. This means that we should skip the first 22 years. We reset X and y to work only with the last years.
skip = 22*12X = X[skip:]y = y[skip:]
Now we can approximate data with a sinusoid. The equation of a sinusoid is A*np.sin(omega*x+phase) + offset, where:
|A| indicates the amplitude, i.e. half the distance between the maximum and minimum values of the function
omega is the number of cycles between 0 and 2pi. The period T is 2pi/omega
phase is the horizontal shift
offset is the vertical shift, i.e. the sum between the minimum peak and the amplitude. Here are some details about offset: https://www.ck12.org/book/ck-12-precalculus-concepts/section/5.4/
For more details about sinusoids, you can give look at: https://mathbitsnotebook.com/Algebra2/TrigGraphs/TGsinusoidal.html.
def sinusoid(x,A,offset,omega,phase): return A*np.sin(omega*x+phase) + offset
We define also initial values for the sinusoid. We set the period T = 12, i.e. the number of months in a year. If the period is not known, we should calculate it. We could use the Fast Fourier Transform. At these links, you can find more details about this aspect: https://stackoverflow.com/questions/16716302/how-do-i-fit-a-sine-curve-to-my-data-with-pylab-and-numpy, https://www.oreilly.com/library/view/elegant-scipy/9781491922927/ch04.html and https://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.fft.html.
T = 12def get_p0(x, y):  A0 = (max(y[0:T]) — min(y[0:T]))/2 offset0 = y[0] phase0 = 0 omega0 = 2.*np.pi/T return [A0, offset0,omega0, phase0]
Now we are ready to approximate our data with a sinusoid. We use the curve_fit() function provided by scipy. We plot results. For more details about the curve_fit() function, please give a look at https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html.
from scipy.optimize import curve_fitimport mathparam, covariance = curve_fit(sinusoid, X, y, p0=get_p0(X,y))step = 1x_ticks = np.arange(skip, skip+len(X)+step+12, step=12*step)x_labels = []x_labels.append(2012)for i in range(1, len(x_tick)+13): x_labels.append(x_labels[i-1]+step)plt.xticks(x_ticks, x_labels)plt.ylabel(‘Number of arrivals’)plt.plot(X, y, color=”red”, linewidth=1,linestyle=’dashed’)plt.plot(X, sinusoid(X, *param), color=”blue”, linewidth=1)plt.show()
We have approximated our data with a sinusoid with a constant value of A. In our case both the amplitude A and the offset are variable. The idea is to calculate all the maximum peaks and minimum peaks in our data. Then we can use their difference to calculate the vector of amplitudes and pass them to the sinusoid() function.
We define the function get_peaks(), which calculates the peaks of an array. We sample the array every T elements and we calculate the peak within each T.
def get_peaks(y, metrics): n = int(math.ceil(len(y)/T)) step = 0 x_peaks = [] y_peaks = [] for i in range(0,n): peak_index = y.index(metrics(y[step:step+T])) x_peaks.append(peak_index + skip) y_peaks.append(y[peak_index]) step = step+T return [x_peaks,y_peaks]
We calculate the maximum and minimum peaks.
min_peaks = get_peaks(y,min)max_peaks = get_peaks(y,max)
Now we can calculate the array of amplitudes and offsets.
A = []offset = []for i in range(0, len(min_peaks[1])): c_a = (max_peaks[1][i] — min_peaks[1][i])/2 c_offset = min_peaks[1][i] + c_a for j in range(0,T): A.append(c_a) offset.append(c_offset)# last two months of 2019 are not availableA = A[:-2]offset = offset[:-2]
Since A and offset are new features, we should add them as variables. Thus we define a new variable features, which defines the new input features of our model. Then we define the function variable_sinusoid(), which takes the features as input.
The only parameters returned by our model are omega and phase thus we modify also the function get_p0() with initial values of omega and phase. The new function is called variable_get_p0().
features = [X, A, offset]def variable_sinusoid(features,omega,phase): x = features[0] A = features[1] offset = features[2] return A*np.sin(omega*x+phase) + offsetdef variable_get_p0(x, y):  phase0 = 0 omega0 = 2.*np.pi/T return [omega0, phase0]
We fit the model and plot results.
param, covariance = curve_fit(variable_sinusoid, features, y, p0=variable_get_p0(X,y))plt.xticks(x_ticks, x_labels)plt.ylabel(‘Number of arrivals’)plt.plot(X, y, color=”red”, linewidth=1,linestyle=’dashed’)plt.plot(X, variable_sinusoid(features, *param), color=”blue”, linewidth=1)plt.show()
The next step involves the prediction of the next sinusoid, i.e. the sinusoid for the year 2020. In order to do it, we need to calculate the next max and minimum peaks. We can do it by approximating the vector of maximum and minimum peaks with two linear regressions, respectively. Then, we can calculate the maximum peak at instant i (August 2020) as the y value of the maximum linear regression given at instant (i-1) + pi (i-1 represents August 2019). Similarly, we can calculate the minimum peak for year 2020.
Firstly, we reshape the peaks to be passed to the linear model. Then, we build two linear models, one for each predicting line, using the sklearn class LinearRegression(). After that, we fit every model using the fit() function.
from sklearn import linear_model# reshape x_peaksx_min_peaks = list(map(lambda el:[el], min_peaks[0])) x_max_peaks = list(map(lambda el:[el], max_peaks[0]))# min modelmodel_min = linear_model.LinearRegression()model_min.fit(x_min_peaks,min_peaks[1])# max modelmodel_max = linear_model.LinearRegression()model_max.fit(x_max_peaks,max_peaks[1])
We append the next peaks both to x_max_peaks and x_min_peaks and then we predict the maximum and minimum peaks through the predict() function.
x_min_peaks.append([x_min_peaks[len(x_min_peaks) -1][0] + T])x_max_peaks.append([x_max_peaks[len(x_max_peaks) -1][0] + T])y_pred_min = model_min.predict(x_min_peaks)y_pred_max = model_max.predict(x_max_peaks)
We plot the two predicting lines.
plt.xticks(x_ticks, x_labels)plt.plot(X, y, color=”red”, linewidth=1,linestyle=’dashed’)plt.scatter(x_min_peaks, y_pred_min, color=”green”, linewidth=1,linestyle=’dashed’)plt.scatter(x_max_peaks, y_pred_max, color=”green”, linewidth=1,linestyle=’dashed’)plt.ylabel(‘Number of arrivals’)plt.grid()plt.show()
We calculate the sinusoid for all the periods, including 2020. We build a variable X_pred, which contains X and the 12 months of 2020.
X_pred = np.array(X)month = X_pred[len(X_pred)-1]for i in range(0,T): X_pred = np.append(X_pred,month) month = month + 1
Now, we can update the variable features_pred to be passed to the variable_sinusoid() function. Firstly, we update the amplitude A and the offset with new values and then we update the features_pred variable.
index = len(max_peaks[0])-1c_a = (max_peaks[1][index] — min_peaks[1][index])/2c_offset = min_peaks[1][index] + c_afor j in range(0,T): A.append(c_a) offset.append(c_offset)features_pred = [X_pred,A,offset]
Finally, we plot graphs.
plt.xticks(x_ticks, x_labels)plt.plot(X, y, color=”red”, linewidth=1,linestyle=’dashed’)plt.plot(X_pred, variable_sinusoid(features_pred, *param), color=”blue”, linewidth=1)plt.grid()plt.ylabel(‘Number of arrivals’)plt.savefig(“data/sinusoid.png”)plt.show()
In this tutorial, I have illustrated a methodology to approximate data having a sinusoidal trend through a variable sinusoid. Then I have described the mechanism to predict the next values of data.
The described methodology can be generalized to all kinds of curves:
load and clean data
plot data to understand the trend line
approximate the trend line with a curve through the curve_fit() function
plot result of approximation and compare with the original data trend line
if needed, improve the approximated trend line
repeat the previous two steps until you get a good approximation
calculate a (linear, polynomial, ...) regression to predict future values
use predicted values to update the approximated trendline.
If you have read this far, for me it is already a lot for today. Thanks! You can read my trending articles at this link.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
The job of the data scientist is very challenging: your knowledge must span from data mining to data analysis up to data visualization. You never stop.
However, what happens when you launch a heavy computation, such as a Machine Learning algorithm on a big dataset? You stop and wait for a result. You may watch your screen for hours, in the hope that your script finishes.
Anyway, this is not the best way to spend your time. You should do something else. You cannot sit down and wait for results.
Continue reading here.
Follow me on Medium
Sign up for my newsletter
Connect on LinkedIn
Follow me on Twitter
Follow me on Facebook
Follow me on Github