Detecting Abnormal Weather Patterns With Data Science Tools | by Chua Chin Hon | Towards Data Science

Around the world, extreme weather events are becoming more intense and frequent.
There’s no missing some of these major weather abnormalities, such as flash floods or prolonged droughts, which can affect large swathes of a country’s population.
But not every weather outlier can be easily observed, particularly in Singapore where the seasonal changes are less obvious to the naked eye. Yet, these “milder” anomalies could be just as important in understanding future changes in the weather pattern.
Data visualisation, such as the range of charts in my earlier Medium post on the subject, provide a quick way to spot such outliers in a dataset. The classic Seaborn pair plot is one way to do this.
But when you have 36 years of weather data, it won’t be easy or efficient to rely on charts to accurately pick out the outliers.
In this third of a multi-part data science project using historical weather data from Singapore, I’ll use Scikit-learn’s Isolation Forest model as well as the PyOD library (Python Outlier Detection) to try to pinpoint outliers in the dataset.
It will be interesting to see the precise dates where these abnormal weather patterns took place. This is also important pre-work for Part IV of the project — time series forecasting, where removal of the outliers would be key to more accurate predictions.
The original source of the weather data in this project can be found here. The Jupyter notebook for this post is here, and you can download the interactive Plotly charts here to explore the outlier data in greater detail.
For the benefit of those who might have no interest in plowing through the detailed data workflow, I’ll start with a quick overview of the results.
The chart below is a slice of the results from two of the PyOD models used to pick out outliers from daily weather in Singapore from January 1 1983 to June 30 2019. I aggregated the results from 4 models in all, and got 816 unique outliers from the dataset comprising 13,330 rows. In other words, outliers comprised about 6% of the data. You can download and explore this subset of weather outliers here.
March 2010, December 1991, and January 1987 tie for the most number of abnormal weather occurrences in a month — 10.
Year  Month    Number of days with outlier weather 2010  3        101987  1        101991  12       101983  4         91992  12        91983  12        82006  12        81998  12        82008  3         71986  1         7
The year 1998 had the most number of abnormal weather occurrences, followed by 1983 and 2013. The highest temperature of 36.0°C was recorded on March 26 1998.
Over the last 36 years in Singapore, December and January were the months with the highest number of days with outlier weather — not surprising given the more intense storms during the end and the beginning of the year.
Some outliers in the weather dataset are more extreme than others, as expected. Plotting the 816 outliers together, we can see a a clear cluster in the centre, bounded by maximum temperatures between 26–34°C and max wind speed of between 20–50kmh. The bigger circles represent days with heavier rainfall.
The more extreme outliers lie to the right, bottom left and top of the chart. Below is a detail of the main clusters in the outliers. Download and explore the interactive version here.
Next, I’ll detail how I arrived at the outlier data and plot a bunch more charts.
Intuitively, we know it doesn’t make sense to try to pick out weather outliers in isolation. For instance, a major storm would obviously result in extreme rainfall, strong winds and lower temperatures.
But I am curious to see what the outlier analysis of a single variable would turn up. I reckon it would be helpful to have some basis of comparison as well with the multivariate PyOD technique.
So let’s see what we can find when the Isolation Forest model is applied to 36 years of daily rainfall data in Singapore. But first, let’s check on the skewness and kurtosis scores of the daily rainfall column.
Skewness: 5.140533Kurtosis: 40.778205
Skewness is a measure of the degree of distortion in a dataset. A symmetrical dataset with a normal distribution will have a skewness value of 0.
Kurtosis is a measure of the outliers in the distribution. See this article here for a reader-friendly explanation.
In the case of the daily rainfall column, we see that the data is highly skewed (skewness score greater than 1) to the right due to days where there are unusual amounts of rain. There’s no rain for about half the time as well (hence the sharp column at 0).
The kurtosis score for the daily rainfall variable is also high (a normal distribution has a kurtosis of three), indicating the presence of a considerable number of outliers.
I ran the rainfall data on the Isolation Forest model, a tree-based model widely used to identify outliers. See these two articles(here and here) for an in-depth explanation.
The key parameter to adjust here is for “contamination”, or the proportion of outliers. I opted for the old default value of 0.1 instead of the new default of “auto”, which would have given me a far higher proportion of outliers.
The main outlier region, in pink, starts from about 19.4mm of rain. In our dataset, there are 1,300 entries that have rainfall higher than 19.4mm. This seems an awfully large number of outliers to try to remove, and I don’t have the domain knowledge to know how best to re-adjust the contamination level.
I repeated the same steps for the daily maximum temperature column, and ended up with a similarly high number of outliers — 1,223. I won’t go into details here but the charts and code are available in my notebook.
Overall, it doesn’t seem right to try to remove such large numbers of outliers in isolation. Let’s see if the PyOD approach is better.
In the Singapore context, the two variables that are most closely associated with abnormal weather would be maximum daily temperature and daily rainfall, making them the obvious choices for outlier detection in PyOD’s models. You can use more than two variables, but doing so will obviously complicate the visualisation of the results.
PyOD has over 20 detection algorithms, many of which you can mix and match together. What I find useful is that the toolkit allows for easy combination of the results from multiple detectors (further details in this paper here).
For the weather dataset, I picked 4 algorithms — the Cluster-based Local Outlier Factor, the Histogram-base Outlier Detection, the Isolation Forest, and the classic K Nearest Neighbors. The results from the 4 models were slightly different, as the charts below show.
I then aggregated the results and filtered out the duplicates via a few simple steps in pandas.
The PyOD library works great out of the box, but the challenge yet again is domain knowledge. Like in the case of the univariate analysis earlier, one key parameter you’ll have to set for PyOD is the contamination/outliers_fraction — the percentage of outliers you expect to find in the dataset.
The default value is 0.1, or 10%, which could be an over or under-estimation. Without better meteorological knowledge, it is hard to say what’s a “correct” outliers fraction to set.
Since I have no expertise in this area, I used a trial-and-error process to see if the outliers identified by the models matched certain key dates where extreme weather, such as flash floods, were reported. Examples include the flash floods in 2010, 2011 and 2018.
Eventually, I settled on an outliers_fraction of 0.04. Anything lower would have missed key dates where abnormal weather patterns had been reported. Anything higher would result in the removal of a very significant proportion of the dataset.
In all, the PyOD models picked out 816 unique outliers out of a total of 13,330 data points, or about 6% of the dataset.
On inspection, the models managed to pick out the heaviest single day of rainfall in the last 36 years — Jan 30 2011 when 216.2mm of rain fell. The PyOD models also successfully picked out the highest daily temperature measured — 36°C, on March 26 1998.
Let’s build on some of the visualisation techniques in notebook2.0 and apply them here for a closer look at the outlier patterns by year, month and year-month combination.
As mentioned in the overview of results earlier, March 2010, December 1991, and January 1987 tie for the most number of abnormal weather occurrences in a month — 10. These are the 10 weather outliers for March 2010:
The maximum temperature and daily rainfall columns clearly show why the weather on these dates are considered outliers. The maximum daily temperatures for March 3, 4, 6, 7 and 9 for instances were very high, at 35°C and higher.
The average daily maximum temperature over the last 36 years is 31.5°C, with a standard deviation of 1.6°C. On the other end of the spectrum, PyOD’s algorithms caught an interesting outlier on March 20 2010, when the daily maximum temperature was just 25.9°C — unusually low by Singapore standards.
March 2010 also saw several days of heavy rainfall, particularly on March 23 (57.6mm) and March 31 (49.1mm). The average daily rainfall over the last 36 years is 5.85mm.
The 10 outliers for Jan 1987 are also obvious at a glance:
Check out the record for the massive downpour of 127.4mm on January 11 1987. That amount of rain in 1 day was about 80% of the median monthly rainfall that Singapore got over the last 36 years (160.1mm).
The Plotly Express interactive charts make it much easier to explore the outliers. I’ve separately plotted the outliers for March 2010 and and Jan 1987.
Unfortunately, Medium doesn’t make it easy for interactive charts to be embedded in the posts, so what you are seeing above and below are screen-caps-gifs. Download the full interactive charts here.
We can also break down the outliers weather data by year, and see which year had the most number of abnormal records. 1998 had the most number of outlier weather occurrences, followed by 1983 and 2013. The highest temperature of 36.0°C was recorded on March 26 1998.
Year  Number of outlier weather days 1998    381983    342013    312006    291984    291986    292010    292007    292008    281988    28
Using a format I had tried in notebook2.0, I picked 3 years with the highest number of abnormal weather patterns — 2013, 1998, and 1983 — and broke them down in a series of matrix panels for easy comparisons.
In the panels below, the larger circles represent days of higher rainfall. The darker and more highly placed a circle is, the higher the maximum temperature and maximum wind speed for that day.
1998 looks to have the most interesting distribution of outliers, with a noticeable stretch of abnormal weather seen from April to September, aside from the heavy rainfall usually seen in January and December.
Some outliers are obviously more extreme than the others. Let’s try to isolate the extreme outliers from the “standard outliers”. In the chart below, the bigger circles signal heavier rainfall. The circles with a darker shade refer to outlier points which are more recent.
We can see a clear cluster in the centre, bounded by maximum temperatures between 26–34°C and max wind speed of between 20–50kmh. The bigger circles represent days with heavier rainfall.
The more extreme outliers lie to the right, bottom left and top of the chart. It’ll be interesting to see if the central cluster moves upward and towards the right in the coming decades.
Analysis of outliers can go in a dozen different directions, depending on the choices and assumptions you make. Share your insights if you use the same dataset for a similar project.
As usual, if you spot any errors, ping me @
Twitter: @chinhon
LinkedIn: www.linkedin.com/in/chuachinhon
And here’s my earlier project on visualising Singapore’s changing weather patterns.