NewYork Taxi demand forecasting with SARIMAX using weather data | by Abdelkader BOUREGAG | Towards Data Science

Taxi demands prediction has become extremely important for taxi-hailing (and e-haling) companies as a way to understand their demand and to optimize their fleet management.
In this article, we will present a method for predicting the number of taxi pickups in a certain region of NewYork. We will perform spatiotemporal time series analysis then we will apply the well-known statistical method SARIMAX on the aggregated data that includes the weather data.
· We will be using NewYork Taxi demand public data available on: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
· Here we will use data from January 2015 to February 2016.
· To quickly download the large Data from NYC dataset website to your google drive account you can follow my tutorial via this link.
· For weather data, we will use data from the Visual Crossing provider: https://visualcrossing.com/weather-data
Time-series forecasting is one of the hot research topics in the field of data science. Various models and approaches have been tested from statistical methods to deep neural networks.
Machine learning enthusiast tend to jump directly to machine learning models when it comes to times series prediction avoiding statistical methods which, in a lot of cases, has shown better performance across both accuracy measures and for all forecasting horizons especially for univariate time series, let’s first get an idea on how to classify a time series problem. Dr Jason Brownlee in his book introduced a framework to classify a times series problem as follows:
Inputs: Historical data provided to the model in order to make a single forecast.
Outputs: Prediction or forecast for a future time step beyond the data provided as input.
Endogenous: Input variables that are influenced by other variables in the system and on which the output variable depends.
Exogenous: Input variables that are not influenced by other variables in the system and on which the output variable depends
Forecast a numerical quantity vs Classify as one of two or more labels.
One or multiple variables measured over time
Forecast the next time step vs forecast more than one future time steps.
fit the model once and used to make predictions vs fit the model on newly available data prior to each prediction
Observations made are uniform vs observations are not uniform.
Now that you got an idea about the type of time series problem we are in front of, let’s introduce the SARIMAX model:
SARIMA stands for (Seasonal Autoregressive Integrated Moving Average) a variation of ARIMA model, a model used in univariate time series prediction, which supports the seasonal component of the series. It supports an autoregressive and a moving average component. It has three hyperparameters to specify the autoregression (AR), differencing (I),and moving average (MA) for the seasonal component of the series, as well as an additional parameter for the period of the seasonality.
X stands for including Exogenous variables.
· p: Trend autoregression order.
· d: Trend difference order.
· q: Trend moving average order.
P: Seasonal autoregressive order.
D: Seasonal difference order.
Q: Seasonal moving average order.
m: The number of time steps for a single seasonal period.
Enough theory, let’s now get into the code !
Data cleaning/preprocessing is a very important step in any machine learning project, yet it’s a tiring step, if you are more interested in applying the SARIMAX method than cleaning the data you can skip this part since I will provide you structured and cleaned data via a drive folder, But it’s important to understand how the data was cleaned and structured.
PS: Find the link to the cleaned data folder down in the section (Cleaned Data folder link)
data_Jan_2015 = dd.read_csv("/content/drive/My Drive/nyc-data/yellow_tripdata_2015-01.csv")data_Jan_2015.head()
We import the data of the first month of 2015 on dask dataframe, the data contains a lot of features, but I will present only the ones we’ll need for cleaning the data:
tpep_pickup_datetime: the date and time of pickup
tpep_dropoff_datetime: the date and time of dropoff
pickup_longitude: longitude of the pickup
pickup_latitude: latitude of the pickup
dropoff_longitude: longitude of the dropoff
dropoff_latitude: latitude of the dropoff
trip_distance: the trip distance
First, we select only the needed columns, calculate the trip_duration and speed to use it in the data cleaning processes.
After preparing the data frame, we will process to remove the erroneous values by analyzing the quartiles graphs of each column:
After printing the quartiles & percentiles we observe the 99.9th percentile of trip distance is 22.58miles, however, 100th percentile value is 15420004.9miles, which is very high. So, we are removing all the data points where trip distance is greater than 23miles.
Doing the same process on other quantitative variables (ِCheck the GitHub repo for code)
By visualizing some of the pickups and dropoff out of NewYork, some are in the ocean, these will be deleted from the data.
After cleaning the demands out of NewYork, we proceed with clustering the demands by pickups using Kmeans algorithm. we divide it by 30 clusters.
#Clustering pickupscoord = new_data_Jan_2015[["pickup_latitude", "pickup_longitude"]].valuesregions = MiniBatchKMeans(n_clusters = 30, batch_size = 10000).fit(coord)cluster_column = regions.predict(new_data_Jan_2015[["pickup_latitude", "pickup_longitude"]])new_data_Jan_2015["pickup_cluster"] = cluster_column
Next, we will group demands by region.
#Grouping the mounthly data by regionnew_data_Jan_2015.rename(columns={'tpep_pickup_datetime':'time','trip_distance':'demand',},inplace=True)new_data_Jan_2015['time'] = pd.to_datetime(new_data_Jan_2015['time'])grouped_new_data_Jan_2015 = new_data_Jan_2015[["pickup_cluster", "time", "demand"]].groupby(by = ["pickup_cluster", "time"]).count()
Let’s visualise the regions:
Finally, we prepare the demands of each cluster by adding weather components and resampling the demands by a time step of 1 hour.
#Cluster examplecluster = grouped_new_data_Jan_2015.loc[0]#Resampling Data in region j into one hour stepcluster = cluster.resample('1h').sum()#Feature Engineeringcluster['Date time'] = cluster.indexcluster['Date time'] = pd.to_datetime(cluster['Date time'].dt.strftime('%Y-%m-%d %H'))cluster['hour']=cluster['Date time'].dt.hourcluster['day']=cluster['Date time'].dt.daycluster['dayofweek']=cluster['Date time'].dt.dayofweek#Merging with weather datadf_merge_col = pd.merge(cluster, weather_data, on='Date time')cluster['temperature'] = df_merge_col.Temperature.valuescluster['wind_speed'] = df_merge_col['Wind Speed'].valuescluster = cluster[['hour','day','dayofweek','temperature','wind_speed','demand']]
The cluster (region) will look like this:
This represents the time serie of demands in one region that we will feed to our model to predict future demands.
Each region will be saved in a separate file.
We will do the same steps to prepare the data of the other months ( preprocessing, cleaning, clustering, adding weather features, saving each region’s demands in the correspondent file)
I am making available the cleaned data via this drive folder,
Here you can find all the NewYork data cleaned and clustered by regions, we will use these files to train and test our model.
Now we will start working on the prediction problem, we remind you that this is a time series forecasting problem, we need to test the stationarity of the time series to be able to apply SARIMAX model.
From the decomposition plot, we can detect a seasonality of one day (24 hours)
#Stationarity testimport statsmodels.tsa.stattools as stsdftest = sts.adfuller(train.iloc[:,:].demand)print('ADF Statistic: %f' % dftest[0])print('p-value: %f' % dftest[1])print('Critical Values:')for key, value in dftest[4].items():  print('\t%s: %.3f' % (key, value))
Augmented Dickey-Fuller test:
Null Hypothesis (H0): It suggests the time series has a unit root, meaning it is non-stationary. It has some time-dependent structure.
Alternate Hypothesis (H1): It suggests the time series does not have a unit root, meaning it is stationary. It does not have a time-dependent structure.
p-value > 0.05: Accept the null hypothesis (H0), the data has a unit root and is non-stationary.
p-value <= 0.05: Reject the null hypothesis (H0), the data does not have a unit root and is stationary.
Here we have the p-value <= 0.05 which means that the data is stationarity.
Now we confirmed that our process is stationary, we will read each regions demands and split the available data between train and test sets. We have data from January 2015 to February 2016.
Train data: From January 2015 to December 2015
Test data: January and February 2016
split_date = pd.Timestamp('2016-01-01')train = cluster_cleaned.loc[:split_date]test =  cluster_cleaned.loc[split_date:]
Visualizing the test set we can observe a strange behaviour of the data around Jan 24. this points will affect the results of the model so we will take this into consideration. Quick google research we can find that this behaviour is due the Bizzard that happened on the week of Jan 24, 2016, in USA and NYC especially that closed the roads which affected the taxi demand.
Source: https://www.nbcnewyork.com/news/local/nyc-new-york-city-blizzard-biggest-ever-january-23-2016/831660/
Now that our data is prepared, let’s get into it, SARIMAX has order and seasonal order hyperparameter that need to be tuned, first will apply hyperparameter tuning on each region demands data to find the best model settings using get_sarima_params function:
Next, we apply sarimax model using this function, note that in the get_sarima_params we specified in the seasonal order 24 as seasonality since we detected 24h seasonality in our time series.
Without forgetting to ignore the test data between 23 and 26 January 2016, due to the erroneous behaviour of the demand caused by the blizzard in NewYork city.
Let’s test the model on the first cluster and evaluate the results with MAE and RMSE metrics.
Now we will use Plotly library to visualize the results of the prediction on test set:
We can see that the SARIMAX model perform well on detecting the seasonality of the time series, and poorly when it comes to detecting the variations ( trends ) values, this can be explained as anomaly detection in some problems.
We can see also the period between 23 and 26 Jan, where the model couldn’t detect the strange behaviour.
Find in this drive folder, NewYork taxi demands data cleaned and arranged into 30 clusters, you can copy this folder into your drive and import it to your notebook and apply directly. Happy hacking!
github.com
I hope that you found this tutorial useful. You started by a motivation for why taxi-hailing and e-halling companies need to predict future demand. This was followed by a taste of the theory behind time series prediction and the SARIMAX model. Then you dived into downloading, exploring and cleaning the demands data. This allowed us better understand the behaviour and construction of a Spatio-temporal prediction problem and to test the seasonality, stationarity of the time series. Finally, we implemented the methods to apply the SARIMAX model on the NewYork data.
To improve the results of the SARIMAX model on your data you can try multiple ways, such as:
Rearrange the data into smaller time steps ( let’s say 10 minutes) in order for the model to better detect the fluctuations.
Tune the SARIMAX model on more values.
Clustering the demands into more regions.
Introduction to Time Series Forecasting With Python Dr.Jason Brownlee
I referred to this repo for some steps of data cleaning.