An Introduction to Time-series Analysis Using Python and Pandas
By: Oscar Arzamendia
Tags: Data Science, Pandas, Time Series Analysis, Featured, Python

So… what’s a time series and what makes it special?

From the initial data exploration, it was clear that we are dealing with what is known as a time series. Time series is just a fancy way of saying we are dealing with data points indexed in time order.

Usually, when dealing with time series we look for some special characteristics in our data to be able to make predictions based on it. Specifically, we look for a time series that is stationary.

Stationarity of a time series

We can say that a time series is stationary when its mean and variance are not a function of time (i.e., they are constant through time).

Stationarity is important because most of the statistical methods to perform analysis and forecasting work on the assumption that the statistical properties (mean, variance, correlation, etc.) of the series are constant in time.

How to test the stationarity of a time series?

Stationarity can be assessed in two ways:

Visually inspect the data points and check how the statistical properties vary in time.

Perform a Dickey-Fuller test.

Let us take a visual approach first and see how it goes.

By plotting the standard deviation and mean along with the original data points, we can see that both of them are somewhat constant in time. However, they seem to follow a cyclical behavior.

Although the visual approach can give us a clue, applying the Dicky-Fuller Test (DF-test) can provide a more precise way to measure the stationarity of our series.

Results of DF-test

I will not go through much detail on how the DF-test work, but let’s say all we need to care about is the numbers we see in “Test Statistic” and “Critical Values”. We always want the former to be less than the latter. And the lesser the value of Test Statistic the better.

Our series is stationary given that the Test Statistic is less than all the Critical Values, though not by much.

In case you ever need it, below goes the code I used to evaluate the stationarity.

def test_stationarity(timeseries):



# Determining rolling statistics

rolmean = timeseries.rolling(12).mean()

rolstd = timeseries.rolling(12).std()



# Plot rolling statistics:

orig = plt.plot(timeseries, color='blue',label='Original')

mean = plt.plot(rolmean, color='red', label='Rolling Mean')

std = plt.plot(rolstd, color='black', label = 'Rolling Std')

plt.legend(loc='best')

plt.title('Rolling Mean & Standard Deviation')

plt.show(block=False)



# Perform Dickey-Fuller test:

print ('Results of Dickey-Fuller Test:')

timeseries = timeseries.iloc[:,0].values

dftest = adfuller(timeseries, autolag='AIC')

dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])

for key,value in dftest[4].items():

dfoutput['Critical Value (%s)'%key] = value

print(dfoutput)

What if our time series was non-stationary?

There are some techniques one can apply to stationarise a time series. The two I am more familiar with are:

Transformation: apply transformation which penalizes higher values more than smaller values. These can be taking a log, square root, cube root, etc. This method helps in reducing the trend.

Differencing: take the difference of the observation at a particular instant with that at the previous point in time. This deals with both trend and seasonality, hence improving stationarity.

Pandas and numpy provide you with very practical ways to apply these techniques.

For the sake of demonstration, I will apply a log transformation to the dataframe.

# Transform the dataframe:

ts_log = np.log(data_df) # Replace infs with NaN

ts_log.replace([np.inf, -np.inf], np.nan, inplace=True) # Remove all the NaN values

ts_log.dropna(inplace=True)

Bonus track: We can even apply a smoothing technique over the transformed data set to remove the noise that may be present. A common smoothing technique is to subtract the Moving Average from the data set. This can be achieved as easy as:

# Get the moving average of the series

moving_avg = ts_log.rolling(12).mean() # 12 months # Subtract the moving average of the log-transformed dataframe

ts_log_moving_avg_diff = ts_log - moving_avg # Remove all the NaN values

ts_log_moving_avg_diff.dropna(inplace=True)

test_stationarity(ts_log_moving_avg_diff)

Clearly, we can see that applying log transformation + moving average smoothing to our original series resulted in a better series; in terms of stationarity.

To apply differencing, Pandas shift() function can be used. In this case, first order differencing was applied using the following code.

ts_log_diff = ts_log - ts_log.shift()

plt.plot(ts_log_diff)

Log-transformed data set after differencing

Let us perform a DF-test on this new resulting series.

ts_log_diff.dropna(inplace=True)

test_stationarity(ts_log_diff)

With the log transformation and differencing the test statistic is significantly smaller than the critical values, therefore this series is too more stationary than the original series.

Wrapping up…

When we face a predictive task that involves a time series, we need to analyze said series and determine whether it is stationary or not. To determine the stationarity, we can either plot the data and visually inspect the mean and other statistical properties or perform a Dickey-Fuller Test and look at the Test Statistic and Critical Values. In case the series happens to be non-stationary, we can apply techniques such as transformation or differencing to stationarise the series.

After all this analysis and preparation, the next step on the project was to forecast with the time series, but that’s a topic for another post :)