Wikipedia Web Traffic Time Series Forecasting-Part 2
By: Tushar Aggarwal
Tags: Time Series Analysis, Time Series Forecasting, Heroku, Flask, Machine Learning

Hello all, this is the 2nd part of our series, the series includes-

In the previous blog, we saw detailed analysis of the data. Now, it’s time to start some feature engineering and building models.

Table of Content-

Baseline Model Feature Engineering Modelling Deployment Future Work References

1. Baseline Model-

First, we will make prediction only based on medians of the data and we will use it as our baseline model.

Now, we will use median of data to make predictions, for each page we have taken median for each weekday and used it to make predictions for the next 62 days accordingly.

Above code will calculate median for each weekday for every page.

We have calculated medians for each page for each weekday, now we will make predictions for Kaggle and make a csv file based on the format for sample_submission file.

After making submission on Kaggle for the above predictions, here are our results.

For this submission, we are currently at 312th position out of 1095 total teams.

This model also performed decent, now it will be a challenge for us to build good features and beat this score by a significant margin.

2. Feature Engineering-

Lets start with some feature engineering !. We will create features for last 15 days of data and through feature engineering techniques we will try to use the rest of the to create features for the last 15 days. As we saw that there was high correlation for 7 days lag, so the first feature we are creating is the week day feature. Other than this , we will create 3 features which has total visitors on the previous 3 days, threedays1 will have traffic for (d-1)+(d-2)+(d-3) days, threedays2 will have traffic for (d-4)+(d-5)+(d-6), similar for threedays3. Another feature will be weekday which contains as the name suggests day of the week.

Above code creates all the features that we have discussed.

As we know that language, type of access and type of client do we have an impact on traffic, so using these as a features can be useful.

lang_feat - It will contain the language feature.

spider_feat - we are adding 1 for spider data and 0 for non spider data.

access_feat - we are adding 0 for all access, 1 for desktop and 2 for mobile.

As we are creating data for last 15 days, so for each page above features will be repeated 15 times.

Now, we will create features based on Fourier transformation. As we saw during analysis that there were some unnecessary peaks at the start, so, after removing some initial values, we will take top 3 peaks and use them as features.

We have created all the features that we have discussed till now, we will use these to train model but we will create some additional features also and we will test if they are any helpful. As we creating features for last 15 days, so through these additional features we will try to use the rest of the data as well.

monthly_trend - This will contain average of medians of traffic on past1,2 and 3 months and these values are also medians of 2 days before and 2 days later during that month.

2. quarterly_trend - Similar to monthly_trend, just work on values on past 120,230 and 340 days values.

3. yearly_trend - This also works as above two feature, just for 1 last year.

Summary and key take aways from our work till now.

First, we loaded the required libraries and files.

Handling Missing values :

We used linear interpolation for filling missing values that were not missing for many consecutive days. For filling consecutive missing values, we used 0.

Data Analysis:

We saw that most of the traffic was on English language pages.

There was a very high correlation for 7 days.

There was very less traffic by bots as compared to actual human.

February has most number of visitors.

Weekdays and weekend doesn’t actually make much difference.

Last three months of the year has more number of visitors on an average than other months of the year.

On average, English language has the most number of visitors.

Feature Engineering :

We used rolling window technique to generate features, from every time series we used 15 days of data. These are the features that we generated-

Day of the week

Number of visitors on d-7 days

Total visitors on last 3 buckets of 3 days.

Language of the page

Client of the page (All access, mobile, desktop)

Spider or actual human traffic

Top three peaks from Fourier transformed data.

We created some extra features also which will be tested to see if they improve any performance. These are the extra features-

Monthly trend (Average of data on 30,60,90 days ago)

Quarterly trend (Average of data on 120,230,340 days ago)

Yearly trend(Average of data from 365 and 730 days ago)

Making data model ready-

Now, we will start to build model based on the features that we have created. First we will convert all the categorical features into one hot encoded features and normalize all continuous feature.

Now, we will split our data into train and test. We will split our data into 75 and 25.

We have already discussed that for evaluating our model, we will be using SMAPE, as there is no inbuilt function for that in python we have to implement that and we will use custom metrics for evaluating our model.

Let’s start with model building.

3. Modelling-

Linear Regression-

These are not good results, linear regression is performing even poor than median submission. We should try more powerful algorithm and see how they performs.

Decision Tree-

0.49,results are good in comparision to Linear Regression but overall these does not seem capable of getting us in good position on leaderboard. Besides, we can not expect that we will be able to produce same results on leaderboard also.

Random Forest-

These results are far better than Decision tree and if we can reproduce the same score on leaderboard then we will be in Top 3 but keep in mind that we are using certain features for this prediction that we don’t have future data and we have to make prediction sequentially in order to make predictions and use these predictions as a feature for future predictions, the errors in our predictions will increase exponentially as we will try to predict for more farther dates. So, we have to get much better score here from the score that we expecting on leaderboard because leaderboard score will greater than this, we just have to be sure that leaderboard score doesn’t get much worse.

XgBoost-

These are very good results but let’s see if we can reproduce these in competition also.

During Feature engineering part we created some extra features to try and check if they can provide better results, now we will use those features as well and train all of the previous models again.

Now, our data is ready with some new features. Now, let’s see how our model performs on this.

smape on DT with best hyperparameter is: 0.38592385

smape on random forest is: 0.3600751026560188

smape on xgboost is 0.3320209381486345

Of all the models, XgBoost performed best but new features did not prove to be much useful.

Making predictions for Kaggle-

Now, we will create all the features for predictions that we created during feature engineering, all the additional features that we created did not prove to be much useful in case of xgboost, so will not use those features for kaggle predictions.

Here are the results-

This is exactly what i was fearing, error indeed increased exponentially as we predicted for more farther dates, this score is worse than our median submission. Now, we have to think of something else.

Deep Learning to the rescue-

Now, we will try LSTM as well. Like before, we will not use any feature that uses predicted data as it will create problems.

As we have more than 145k time series, we are using only 50% of data to train the model.

For evaluating our model, we will use Mean Absolute Error(MAE) on log1p of data. This is pretty similar to SMAPE on original data. After final predictions we will use expm1 on data to convert it back to original form.

Along with the time series, we will use the three features as well that we can extract from the page name

Above code creates all the features that we need for our model, now we will label encode all the categorical features , reshape them in proper shapes to fed it to the model and split data into train and test.

Now, our data is ready and we can start building our model architecture. Model will have 4 input layers(one for each feature), time series data will go through LSTM layer and all the label encoded features will go through an embedding layer and then we will flatten the output so that we can concatenate all these layers . After that we will use dense layer and an output layer.

Code for above discussed model will look like this-

Now, let’s take a look at the architectural graph of our model-

After training the above model, performance looks like this for the last few epochs-

Here is the performance of the model over all the epochs-

Blue line represents loss on validation data while Orange represents loss on training data.

As we have already discussed that MAE on log1p of data is somewhat similar to SMAPE, so we should get pretty good score on leaderboard also.

After making predictions and submitting the file on Kaggle, here are our results-

The score is good and it takes us in top 5% on leaderboard and 53/1095 position.

Comparision of all the models that we tried-

We can see that Tree based model was unable to perform good, their scores were worse than median model but LSTM proved to be a good choice maybe the reason is that they are specialised in seq-seq prediction.

4. Deployment-

What good a model is if it doesn’t reach customers. Most of the case study blogs skip this part but we will leave no stone unturned. We will be using flask for deployment and Heroku for service.

In the modelling section, we were training models in such a way that they will predict for the next 64 days but here we need prediction just for one specified date. So, i have trained LSTM in such a way that it takes data for last 5 days as input instead of 200 days and predict for next day. The code would be pretty similar but in case you still want to see my code you can check my github repository. Now, we have to build a pipeline for the preprocessing, feature engineering and prediction. We can’t ask user to enter the traffic for last 5 days and the page name itself to make prediction. So, we would be taking index and date as input, index will correspond to a particular page name as mentioned in the dataset and according to the date for which he wants to make prediction data will be fetched from the csv file. Heroku gives us a limit of 500 MB and the Tensorflow module alone will take more than 300 MB so we can’t use our complete dataset for that. So, instead of 145k pages we will be using 10k pages only for model deployment.

Let’s get started with the pipeline!

Output will look like this-

Ok, so now we are done with the pipeline and now it’s time to use flask to deploy and we also have to build HTML pages( I will keep it very simple).

Let’s understand what this code actually does.

At first we are importing all libraries and data pipeline from a python file. After that we are defining different app route in order to render different HTML pages, in the predict function we are getting input from the HTML page, after getting these input, it’s all about the data pipeline that we have already discussed.

Now, lets proceed to Heroku. So, Heroku is free and pretty easy to use, we will use Github to deploy our model on Heroku. So basically there are two extra files that we have create Procfile(without any extension) and Requirements.txt

Procfile contains the name of the file that needs to be run first and requirements.txt contains names of all the modules that needs to be installed. Now, we just have to choose ‘connect to github’ on Heroku and search our github repository and click deploy. If all the code is correct, model will be deployed and a link will be generated for your app.

After doing all the above steps, here is our app. It may take some time to open the link, have patience.

Here is one video showing the running demo of our project.

5. Future Work-

Different type of smoothing and transformation of data can be tried in order to improve the results. Attention mechanism based models can be tried in order to use information from distant past to improve the results. As we discussed in Existing approaches, there are few simple statistics based method which proved pretty good, these methods can be used to provide extra features to the models to improve the results.

6.References-

2. https://www.kaggle.com/c/web-traffic-time-series- forecasting/discussion/43727

3. https://cs.adelaide.edu.au/~zbyszek/Papers/p51.pdf

4. https://blog.octo.com/en/time-series-features-extraction-using-fourier-and-wavelet-transforms-on-ecg-data/

5. https://www.youtube.com/watch?v=mrExsjcvF4o&t=830s

6. https://www.appliedaicourse.com/course/11/Applied-Machine-learning-course

If you have any query, feel free to comment down or you can also contact me on Linkedin. You can find my complete project here.