How I used pandas and datetime to improve accuracy in a traffic time series problem
Tags: Time Series Analysis, Artificial Analysis, Datetime, Python, Pandas

How I used pandas and datetime to improve accuracy in a traffic time series problem Tracyrenee Follow Dec 23, 2020 · 5 min read

During the course of my studies in time series analysis, I have discovered that there are some time series datasets that perform better using the more traditional methods of machine learning because of the way the data is presented. For example, I have seen some datasets that, although time series, were depicted in such a way as to nullify the components of the series itself.

The dataset I am presenting in this post is an example of a data science problem where it is difficult to use models, such as Facebook Prophet, which are designed specifically to predict on time series problems.

I have selected a traffic dataset from Analytics Vidhya and the problem statement and link to the datasets can be found here: McKinsey Analytics Online Hackathon (analyticsvidhya.com)

The problem statement for this traffic problem reads as follows:

“You are working with the government to transform your city into a smart city. The vision is to convert it into a digital and intelligent city to improve the efficiency of services for the citizens. One of the problems faced by the government is traffic. You are a data scientist working to manage the traffic of the city better and to provide input on infrastructure planning for the future.

The government wants to implement a robust traffic system for the city by being prepared for traffic peaks. They want to understand the traffic patterns of the four junctions of the city. Traffic patterns on holidays, as well as on various other occasions during the year, differ from normal working days. This is important to take into account for your forecasting.

To predict traffic patterns in each of these four junctions for the next 4 months.

The sensors on each of these junctions were collecting data at different times, hence you will see traffic data from different time periods. To add to the complexity, some of the junctions have provided limited or sparse data requiring thoughtfulness when creating future projections. Depending upon the historical data of 20 months, the government is looking to you to deliver accurate traffic projections for the coming four months. Your algorithm will become the foundation of a larger transformation to make your city smart and intelligent.”

The program to solve this data science problem has been written as a .ipynb file in Google Colab because it is a free online Jupyter Notebook that can be accessed from computers that have internet access and Google connectivitivity.

The Python libraries are already installed on Google Colab, so I only needed to import the pre-installed libraries.

I then read in the datasets from my personal GitHub account, where I had stored them when I downloaded them from the Analytics Vidhya problem page:

I checked for any null values and in this instance there were none that needed to be imputed:

In the code below I used pandas and datetime to extract the hour, day of the week, week of the year, month, and year to categorise the data so that this information would be better utilised during the training of the model:

I then created the variables, ID_train and ID_test, which are used to store the data from train.ID and test.ID. Once these two variables had been created, I dropped the “ID” column from both train and test because they are not needed during the training process:

I put the time series data on a graph and there it can be seen visually how the time series data is not linear and therefore time series models will not give the best predictive results:

I defined the X, y and X_test variables, which need to be established so the data can be split and used during the modelling process. The target variable, y, contains the data from train.Vehicles, which was then dropped from the train dataset. The independent variables, housed in the train and test datasets, were used to develop the X and X_test variables by additionally dropping the DateTime column.

Once the X dataset had been developed from the train dataset, it was used as input to sklearn’s train_test_split() function to split the data into training and validation datasets.

I then used sklearn’s make_pipeline() function where StandardScaler() and the model were activated. I used BaggingRegressor() because I have had superior results from this ensemble model, achieving 99.11% accuracy:

I used the model to predict on the validation set and achieved 96.88% accuracy on the predictions made on this dataset:

I put the predictions made by the model onto a graph and visually compared the predictions against the validation target:

I then used the model to predict on the test set. I created a dataframe to house the predictions and converted this dataframe to a .csv file:

When I put the predictions on the solution checker I obtained a root mean square error, RMSE, of 9.14, which is a very respectable reading indeed. Because the predictions I obtained were of a high accuracy and low error, I also posted the code on the Analytics Vidhya website:

The code for this post can be found in its entirety in my personal GitHub account, the link being here: Traffic-Forecasting/TS_Traffic_Forecast_Bagging.ipynb at main · TracyRenee61/Traffic-Forecasting (github.com)