Time Series Prediction using Spark
By: Akash Goyal
Tags: Machine Learning, Time Series Model, Time Series Analysis, Apache Spark, Python

Time Series Prediction using Spark

Photo credits Apache Spark Foundation

Introduction to the Objective

These days in high-tech or smart cities the pedestrian counts can be monitored by deploying sensors at certain locations which can count the number of pedestrians every hour(as per the data used for this blog) or as required. From the title of the post itself one can understand that here we will try to predict the count of pedestrians or pedestrian traffic at certain locations for the next hour from the data of previous hour(s). This technique is also called a one-step time-series prediction, where we are predicting the next value with the previous values. Therefore this is a time-series regression type of problem as the data to predict is of continuous nature. Using these predictions we can select the locations with most traffic which can then be used by certain companies to market their products, performers in the music and entertainment industry to make sure that they are heard by the most amount of people, etc.

The above statement clears the objective but at the same time gives room to some basic questions that can come to the mind of an individual. Let’s first answer those questions:

Which tools would be used to perform the time-series prediction?

As per this blog the tool used would be Apache Spark. The programming language used would be Python. You can go to my github page to get the complete code.

What type of data would be used in this blog?

There would be two datasets used in this blog

Pedestrian Counts data

Sensor Information data

Unfortunately the actual data used cannot be uploaded because of copyright issues but at the same time I have shared the structure of the data so any data of similar structure or that can be converted into a similar structure can be used.

Now that we have addressed the basic questions and objective, let’s dive into achieving it.

Flow used to achieve the objective

Steps

Data Loading Data Exploration Feature Extraction and Transformation Model Building and Persisting

Data Loading

To load the data first we need to create the schema of the data to be read. Like below.

The StructType function can be used to define the structure and the StructField function can be used to add each attribute with the corresponding data-type.

Once the schema is created we can then easily load the data using spark’s spark.read.format('csv') function as the source data is in comma-separated format, further the schema() function can be used to provide the schema(created above) as the parameter to read the data. Other options can be used for example to specify the timestamp format, etc. Have a look at the code below.

If you want to have a look at the resultant data structure to make sure that the data is showing correct schema it can be checked using printSchema() function. See the below code and output.

Data Exploration

Now comes the part where we need to explore the data to see certain trends/patterns which would help to choose the features to be used to build the machine learning model.

As the data has been loaded in spark dataframes, which are not that easy to directly plot with therefore, the required data has been converted into Python’s pandas dataframe for plotting.

We will first plot a histogram to see the distribution of the Hourly_Counts column using the code below.

Histogram plot for the Hourly_Counts column

This is how the histogram plot would look like, obviously depending on the distribution of your data the histogram can change. Please note that for this plot the scale has been changed to log scale to visualize it in a better manner.

Many other visualizations can be created after converting the required data into pandas dataframe. Please refer to the code to see what other visualizations can be created.

Feature Extraction and Transformation

We will have to create the Previous_Hour_Counts column from the Hourly_Counts column using the windowing function called lag() . We have create a lag on the Hourly_Counts column, partition over the Sensord_ID column and order on the Date_Time column to make sure that the count value of a sensor is not passed to the other sensor while creating a lag. Every cell value of Hourly_Counts column will be shifted one row up and the column would be named Previous_Hour_Counts . Below is the code and an animated example of how the data in Previous_Hour_Counts column would look like.

Please make sure to remove the rows where Previous_Hour_Counts is NULL as those rows would not be able to contribute to the prediction.

We also need to remove the unwanted data for better efficiency. We can filter the data only between say 2016 to 2018 and only for hours between 9am to 11pm both inclusive, which would make sense because before this time the general notion says that the pedestrian count would be low as hours between 11pm to 8am are ‘unreasonable’ for a lot of people to be walking on the streets.

Looking at choosing the features

ID: This column is a unique identifier for each row and therefore holds no importance from prediction’s standpoint and therefore need not be used for prediction.

Year, Month, Date, Time: These column have would prove to be be very helpful for prediction as per the exploration(please check the github page for code). They look very important as they are showing patterns which can be learnt by the machine learning algorithm and hence provide some help with the prediction.

Sensor_ID: This feature would also provide a lot of information because there would be some sensors/locations which might always receive high Pedestrian counts and there might be some others which receive low counts. Therefore, this feature can show patterns.

Hourly_Counts: This column would be present for training the models for regression.

Previous_Hour_Counts: As it can be seen from the visualizations the Hourly_Counts column can be a very good feature to predict the next value from the historical value and therefore we can use that as a feature for prediction.

The columns ‘Date_Time’, ‘MonthNum’, ‘DayofWeek’, ‘Mdate’, ‘Sensor_Name’ need to be dropped as they do not add much value to the model or their categorical counterparts have already been added.

We need to convert the datatypes of certain columns to the datatypes we need, Time to StringType(), Previous_Hour_Counts to IntegerType(), Hourly_Counts to IntegerType(), Sensor_ID to StringType().

The column name of Hourly_Counts can be changed to target (This is optional) using the below code.

Next we need to use the String Indexing technique to encode the string/categorical columns to label indices. Now the One-Hot Encoding technique can be used for these indexed values(from StringIndexer) so that the model can use these indexed values as features. Using One-Hot Encoding these values are being encoded in this manner as their ordinal values would not have any importance and may affect the model negatively. To use the above created features we need to assemble them into vectors using Vector Assembler so that they can be used by the models. Let’s create stages/steps for these processes to be able to use them later in the model building phase.

Model Building and Persisting

In this part we would create the machine learning model with the training data and then calculate metrics on that model using the test data for model assessment. We will be using a basic Decision Tree model. Let’s create the model building stage now. For other models have a look at the code.

Now using the Spark’s ML Pipeline API we can use these stages to create a pipeline. One of the main advantages of using the pipeline API is that it makes the code more manageable and readable.

The next part is to split the data into training and testing and then using the training data to build the model and to get the predictions for the test data.

Now the metrics can be calculated to assess the model, in this post RMSE and R-squared value has been used. There are many more metrics which can be used based on your choice and use-case.

To save/persist the model we can use the below code.

The above code is used because this model is a pipeline model and first we need to extract the model and then save it and stages[-1] helps us to get the model object which can then be saved using save() function. We can also save the pipeline model directly by using save() function without stages[-1] .

Do not miss the next post which is in continuation to this one where you can how to actually apply the model to streaming data and also visualize it.

Please take a look at the github page for much more things that I have done in the python notebook, had to cut-short in the post to be concise.

Thank you so much for taking time to read the post, kindly leave comments if you want to discuss anything related to this post, I will try my best to help you out.