HDSC Stage F OSP- Weather Forecasting: A Time Series Analysis
By: Iheagwara Ifeanyi Benny
Tags: Forecasting, Hamoye Features, Hamoye Latest, Weather, Lstm

Introduction

Time series is a stretch of values on the same scale indexed by a time that occurs naturally in many application areas such as environmental, economics and finance, and medicine. The aim of time series analysis is to formulate time series data in order to gain insight into past and likely future events.

During Stage E of the Hamoye Data Science Internship program, my group and I worked on data taken from the United States National Oceanic and Atmospheric Administration National Centers for Environmental Information website. This dataset was originally hosted on Kaggle to assist in the analysis of the Aerial bombing operations dataset. It contains information on weather conditions recorded each day at various weather stations around the world. Information includes precipitation, snowfall, temperatures, wind speed, and whether the day included thunderstorms or other poor weather conditions.

Multivariate models like VAR, VECM, and LSTM are often used to model evolution in times of economic issues. In this project, we demonstrated that this model is also valuable in the environmental field, where the evolution of climate change is causing many concerns.

In this article, we would be sharing the approach we took in using these models to forecast the minimum, maximum, and mean temperature in the city of Lagos, Nigeria. The dataset used for the project can be found here.

Getting a glimpse of the data: Exploratory Data Analysis

The link contains 1940–1945 data for 162 stations outside of the United States. The actual period of data availability varies depending upon the station’s activity. Many stations in the European and Pacific theaters of operation are included. All data are a summary for the calendar day.

Data Preprocessing

For the weather condition CSV file, we dropped columns that were more than 80% empty.

total = weatherdf.isnull().sum().sort_values(ascending=False)

percent = weatherdf.isnull().sum()/weatherdf.isnull().count()).sort_values(ascending=False)

missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])

f, ax = plt.subplots(figsize=(15, 6))

plt.xticks(rotation='90')

Next, we dropped the “TSHDSBRSGF” column which we learned from the dataset description was the same as the “PoorWeather” columns. Then, we dropped “MAX”, “MIN” and “MEA”, these columns were the recorded temperatures at the given time (in Fahrenheit), but we made use of the “MeanTemp”, “MaxTemp” and “MinTemp” in Celsius. The next approach was to drop the “PRCP” and “SNF” which were duplicates of the “Precip” (precipitation) and “Snowfall” columns from the dataset description. After all these, we were left with “Snowfall”, which had 10% nan and our “PoorWeather” which had 70% nan.

Since we are dealing with a time series (a series of dataset which are measured and tracked over time), I wouldn’t advise anyone to drop nan. Resolving the issue of nan in the “Snowfall” columns, we used the “bfill” since the weather of today is logically related to the day before.

This leaves us with the “PoorWeather” which is related to the “Snowfall”. According to the description, the weather is said to be poor if there is snowfall, hail, or thunder and would be recorded as 1, otherwise, it is recorded as 0.

#setting STA as the index to select APAPA weather station df_weather = df_weather.loc['30001']

df_weather = df_weather.set_index('datetime')

df_weather.head()

We made use of this new data frame to define our model.

Time to Build our Multivariate Time-Series Model

Multivariate time series has more than one time-dependent variable. Each variable depends not only on its past values but also has some dependency on other variables. We visualized the time series to give us the necessary intuition needed for model evaluation.

After necessary cleaning & pre-processing (filling the missing values with previous ones), we finally have the Apapa weather station for necessary analysis

Stationarity Check

We know from studying the univariate concept that a stationary time series will more often than not give us a better set of predictions