Predicting Dengue Spread Using Seasonal ARIMAX Model and Meteorological Data | by Nam Nguyen | Towards Data Science

During the last decades, dengue viruses have spread throughout the sub-tropical parts of the world, with an increase in the number of severe forms of dengue such as severe bleeding, low blood pressure, and even death. Because the disease is carried by mosquitoes, the transmission of Dengue fever is related to meteorological variables such as temperature and precipitation. In this blog post, I will go over how I tackle the DengAI competition Paper and Rcode can be found on my github!
Here is the game plan:
Data SplitDescriptive Analysis and Data VisualizationHandling Missing ValuesModel DiagnosticsModel FormationForecasting
Data Split
Descriptive Analysis and Data Visualization
Handling Missing Values
Model Diagnostics
Model Formation
Forecasting
The DengAI competition data set is essentially a messy dataset with two different cities and two different time frames. I split the dataset into two small datasets based on two cities: San Juan and Iquitos. From two datasets, I divided each into train and validation sets in order to test different models before choosing the best one to perform on the final test set.
The Data Processing for both San Juan and Iquitos were quite similar, so I will only show San Juan’s graphs in this section. Charts and Graphs of Iquitos will be provided in my paper.
Each dataset contains 25 variables including meteorological and geographical variables. See the details about each variable here.
A Correlation plot will be helpful to diagnose if there are any linear relationships between 2 variables. We can see none of those variables is highly correlated to total cases (Figure 3). However, if we include more than 2 variables and run a multivariate linear regression, the magnitude of each predictor might change.
The time series plot indicates that there might be seasonality in the data since the number of total cases peaks every year around summer time. Two scatter plots also suggest possible linear relationships between total cases and two climate variables. (temperature and humidity)
Looking at the dual time series plots of total cases and temperature, and total cases and humidity, we can see sometimes the number of cases and climate values move up and down together; sometimes the weather variables peak or plummet first, then the total cases go up and down in 3 -4 weeks later. More interestingly, Dengue symptoms usually start 4 to 7 days after you are bitten by an infected mosquito. Mosquito might go through its life cycle in 14 days at 70° F and take only 10 days at 80° F. Therefore, in average, it takes 3–4 weeks for the cases to be reported after the temperature, precipitation or humidity go up and down.
The Descriptive Analysis and Data Visualization suggest a time series model with additional predictor variables, and possibly some previous lags of the predictors.
I handled missing data by using Last Observation Carried Forward method in R, which is basically replacing each NA with the most recent non-NA prior to it. Here I am assuming that the weather of the current week would not be too different to the weather of the previous week.
# build a function to impute data with the most recent that is non missing  na.locf.data.frame <-     function(object, ...) replace(object, TRUE, lapply(object, na.locf, ...))    #fill in NAs - SJ  sjdata.train.imputed <- na.locf.data.frame(sjdata.train)    #fill in NAs - IQ  iqdata.train.imputed <- na.locf.data.frame(iqdata.train)
An Augmented Dickey-Fuller Test suggests that the time series dataset is stationary. The dataset has complex seasonality with weekly data points (frequency = 52.18). Most of the methods such as ETS, Decomposition require the seasonal period to be an integer. Even if we round it up to 52, most of the methods will not handle such a large seasonal period efficiently. A possible approach could be to use a dynamic harmonic regression model by including Fourier terms in ARIMA model.
Next, to pick MA and AR terms, I first examined the early lags (1, 2, 3, ...) to judge non-seasonal terms. Those spikes in the ACF at low lags indicate non-seasonal MA terms. Additionally, the spike in the PACF at low lags indicates a possible non-seasonal AR term.
With the empirical review and data diagnostics, I decided to try 3 models: SARIMA, SARIMAX, and Neural Networks. I started with a basic and simple Seasonal ARIMA with only total cases, then I added more predictors such as temperature, humidity, precipitation into the model. Lastly, I built Neural Networks model (multilayer feed-forward network) with NNETAR package in R. Here I will report only the process of building the best model, which is Seasonal ARIMAX.
Seasonal ARIMAX
Here after doing researches about what factors affect the proliferation of mosquitos, I added on these variables to Seasonal ARIMA using xreg arguments:
precipitation_amt_mm
reanalysis_dew_point_temp_k
reanalysis_relative_humidity_percent
station_avg_temp_c
station_diur_temp_rng_c
station_max_temp_c
station_min_temp_c
Basically, if you grew up in a tropical country, you might know that mosquitos’ lifecycle includes four separate and distinct stages in both water and air. Therefore, precipitation, temperature and humidity play an important role in accelerating the mosquito population. Refer this link for more information.
Sometimes, the impact of a predictor which is included in a regression model will not be simple and immediate. For example, the high temperature and high quantity of rainfall may impact total cases for some time after a few weeks. I ran CCF (Cross-Correlation Function) to see if any previous lags of the predictors are highly correlated with the current lag of total cases. It seems like the dew temperature of the previous 4 weeks might be positively correlated with the current number of cases.
The selection of ARIMA models was conducted using Akaike’s information criterion (AIC). Of all the model tested, a SARIMAX (3,0,2)(0,0,1)[52] are the most appropriate model for San Juan city, while ARIMAX (0,1,2) with K = 1 for seasonal Fourier term is the best model for Iquitos city.
The final choice between ARIMA and NNET based on in-sample cross-validation using MAE as the main parameter.
Here are the forecast plots of both cities. I combined the prediction of total cases of San Juan and Iquitos and submitted the result to Data Driven website. I got 25.6587 for MAE and ranked 674/4392.
The final model has some shortcomings that can be addressed in the future. Firstly, I totally left out “ndvi” (geographical variables and Satellite vegetation index) while these variables might be important to the breakouts of the disease. However, to include those variables, I need to do more researches to have a better understanding of what, why, and how they matter. Secondly, the model does not fully capture all the factors contributing to the spread of Dengue. More variables related to the infrastructure and healthcare systems of both cities should be included in the model. Lastly, a multiple-seasonal period model could be conducted to show the different seasonal patterns of week, month and year as the current model only captures the weekly seasonality.