Most useful Python functions for Time Series Analysis | by Sivakar Sivarajah | Towards Data Science

These type of data are sequentially ordered data over time and these observations are typically collected at regular intervals, this could be:
Every Second/Minute/Hour
Daily
Monthly
Quarterly/Yearly
Some Real-life Examples are:
Monthly Sales Data
Stock Market prices
Hourly Weather Data/Wind Speed
IoT(Internet of things) Sensors in Industries and Smart Devices
Energy Forecasting
Trends show the general tendency of the data to increase or decrease during a long period of time. Generally, a trend is a smooth, general, long-term, average tendency.
Seasonality: refers to periodic fluctuations. For example, electricity consumption is high during the day and low during the night, similarly online sales go up during Christmas before going down again.
Stationarity is a key characteristic of the time series. A time series is said to be stationary if its statistical properties do not change over time. In other words, it has constant mean and variance, and covariance is independent of time. i.e. A Stock price is not a stationary series, since we might see a growing or decreasing trend and its volatility might increase over time(meaning that variance is changing).
Ideally, we want to have a stationary time series for modelling. In real-world scenarios, not all of them are stationary, but we can make different transformations to make them stationary.
I will use The Daily Delhi Climate recording going over the span from 2013 to 2017. The dataset has got 4 features but for this tutorial, I am going to specifically use only the Humidity Column.
As we can see data from the plot above the data looks stationary and there are few ways to check that! let's look at them
ADF test is the most commonly used test the Stationarity of the series, where the null hypothesis is the time series possesses a unit root and is non-stationary. So if the p-Value in ADH test is less than the significance level (0.05) we reject the null hypothesis.
Null Hypothesis: The series has a unit root (value of a =1)
Alternate Hypothesis: The series has no unit root.
KPSS((Kwiatkowski-Phillips-Schmidt-Shin) test is another test for checking the stationarity of a time series.
However, the null and alternate hypothesis for the KPSS test are opposite that of the ADF test, which often creates confusion. (You could see in the code below the conditions are opposite)
Null Hypothesis: The process is trend stationery.
Alternate Hypothesis: The series has a unit root (series is not stationary).
It’s always better to apply both the tests, so that we are sure that the series is truly stationary.
Autocorrelation is important because it can help us discover patterns in our time series, successfully select the best modelling algorithm, correctly evaluate the effectiveness of our model. Specifically, autocorrelation and partial autocorrelation plots are heavily used to summarize the strength and relationship within observations in a time series with observations at prior time steps.
When there is a strong seasonal pattern, we can see in the ACF plot usually defined repeated spikes at the multiples of the seasonal window. For instance in most “monthly sales time series ” you should be able to see spikes at around at 12th, 24th, 36th.. lines explaining the rise of sales with the advent of Christmas holidays.
Any time series may be split into the following components:
Base Level
Trend
Seasonality
Error
However, It is not mandatory that all-time series must have a trend and/or seasonality. A time series may not have a distinct trend but have a seasonality. The opposite can also be true.[1]
Depending on the nature of the trend and seasonality, a time series can be modelled as an additive or multiplicative, wherein, each observation in the series can be expressed as either a sum or a product of the components:
# Actual Values = Addition of (Seasonality + Trend + Residual)
The df.resample() function let us resample our time series to our desired frequencies: A number of string aliases are given to use common time series frequencies, these are referred as offset aliases. Let’s have a look at a few resamplings using these offset aliases.
Calculates the alegbric difference between one Dataframe element with another element, the default is one meaning it “differences” the element with the prevoius element. Here there is an example with our humidity coloumn
As shown above df[:5]. diff(2) takes the difference between the first and third element of the column, consequently the second and fourth element, and so on.
Calculate the Percentage change between the current and a prior element.
Computes the percentage change from the immediately previous row by default. This is useful in comparing the percentage of change in a time series of elements.
Now let's try to calculate the average mean over all the months, years and day of the week. We will apply pct_change to see how the pattern changed compared to the previous.
Humidity means over all the months, years and day of the week
Note: there is an anomaly in the 2017(“year”) humidity mean! The 2017 data have only one record! therefore this anomaly.
fig, (ax1, ax2,ax3) = plt.subplots(3,figsize=(30,30))ax1.plot(week_mean_df.Percentage_increase,marker='o', linestyle='--', linewidth=5,markersize=30, color='g')ax1.set_title('Weekly Pct_change');ax2.plot(month_mean_df.Percentage_increase,marker='o', linestyle='--', linewidth=5,markersize=30, color='r')ax2.set_title('Monthly Pct_change ');ax3.plot(year_mean_df.Percentage_increase,marker='o', linestyle='--', linewidth=5,markersize=30, color='b')ax3.set_title('Yearly Pct_change');
The shift function shifts the data before or after the specified amount of time.
Shift index by desired number of periods with an optional time freq. If not specified by default it will shift the data by one day as shown above.
if for instance, we want to shift the data by 365 days we could do:
df[:5]shift(365)
Provide rolling window calculations or i.e Moving average calculations
Moving Average is doing the mathematical average of a rolling window of defined width. You should choose the window-width wisely, a large window-size will over-smooth the series. A window-size bigger than or equal to the seasonal duration will effectively nullify the seasonal effect.
In the figure below you can see clearly that a window of less than five is likely to be the threshold as we go up with the bigger window sizes, the series is over-smoothed.
When choosing the window you have the option to test and chose a different type of Window, Below there is an example with a few types.
This provide exponential weighted (EW) functions, weighing more the current values rather than historic values.
In this post, I have tried to explain most of the useful pandas and statistical testing functions for time series analysis. If you could run all the code above, you should have some knowledge on how to perform an analysis on a time series use-case. It will take a lot of practice to become proficient at using all these functions! If you already know these functions I am happy to have refreshed your memory and skills.
Thanks for reaching until here, if you want to learn more about modelling, analyzing and much more on time series, I highly recommend Youtube channel: AI engineering (by Srivatsan Srinivasan) playlist on Time Series, the latter is rich with a lot of quality information, check it out :)
More Reading:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
References:
[1] ML+. 2020. Time Series Analysis In Python — A Comprehensive Guide With Examples — ML+. [online] Available at:https://www.machinelearningplus.com/time-series/time-series-analysis-python/ .