Data science for weather forecast: how to prove a funny theory
By: Julien Emery
Tags: Weather, Technology, Business Intelligence, Data Science, Data Visualization

What are we trying to do?

Before describing what this experiment is all about, I need to give you some context. My colleague Aouss Sbai (co-author of this article) and I were looking for a fun project to work on. So we asked our mentor Plamen Nedeltchev (Distinguished Engineer at Cisco) if he had anything in stock and he shared with us that he had a theory about the weather in San Jose, CA. He told us that he was able to predict if the summer was going to be hot or not solely based on the temperatures of the 19th, 20th and 21st of May. He asked us to prove it.

You read that correctly, predict the average weather of 3 months based on 3 days. Frankly, we did not really believe in it and approached this task with great scepticism. Nevertheless, we got started and tried to understand how we could go about proving such an original statement.

Data collection

The first hurdle

In any data science problem, the starting point of anything is data.

What data do we want exactly? Remember our objective: predict the average summer temperature following the temperatures of 3 days in May (19th, 20th and 21st) for the city of San Jose.

So we started looking for databases or archives of historical temperature data. Guess what? There were none that were either complete enough or simply available to us 😬. The only source of information we found was on a website, the old Farmer’s Almanac, which listed the average temperatures of each day since 1945. Amazing, the job is done then! Well, not exactly…

This is the page corresponding to the 23rd of March, 2019. We have access to the mean temperature, which is exactly what we need. But when it comes to calculating the average temperature of the summer of each year since 1945, this becomes much more tedious (~4500 days). There was no way we would visit a different page for each day and manually gather this data in an excel sheet…

So we decided to automate this task and write a script for it! 🤖

Automation with a web scraping script

Basically, the idea is that the script visits each page independently and looks for the data we want, calculates the average temperature of the summer, captures the temperatures of the 3 days in May, and repeats the process for each year starting from 1945. But how exactly can we do that?

This is the URL of the page you just saw above. As you can see, it is specified in the URL the city and the date you want to access. So we could tell the script which URL to visit, for each day we were interested in. But here comes another issue. Once the script is on the page, how does it detect the temperature that we want?

Well, as you might know, each webpage is written in HTML format, which means that each element that you see on screen belongs to a specific HTML tag. And, luckily for us, each page of that website was structured in the exact same way. So the only thing that we needed to do is identify in which HTML tag was the daily mean temperature stored and tell the script to fetch that specific value. (For those interested, we used the python library Beautiful Soup)

The script was then able to do all the nasty calculations for us and return for each year the average summer temperature and the individual temperatures of the 3 days of May, all bundled in a nice Excel sheet 📝.

what our dataset looks like now (temperatures are in Fahrenheit). The first 3 columns are the 3 days of May, and the last one is the average summer temperature

But (there’s always a “but”), that was not enough. In fact, when you think about it, each line of our Excel sheet represented 1 year (average temp of summer + 3 days of May). So, even if we went back to 1945, that represented only 73 lines… which is far too little data to pretend to do any sort of reliable analysis or prediction ( a couple hundreds would be much better). So we decided to repeat the exact same process for 4 other cities of northern California around San Jose which were subject to the same type of weather but were far enough not to have redundant data (taking San Francisco for instance, which is by the sea would have biased everything, and taking Milpitas, which is in San Jose suburbs wouldn’t have added any relevant data). We now have 370 measurements, which is not ideal, but sufficient to start doing some analysis.

Let the analysis begin!

Data Transformation

Now let’s try to simplify our dataset to make it easier to analyze.

To start things off, we pulled the Excel file data into Alteryx, a data science tool to create end-to-end data pipelines. This will help us prepare and analyze the data all along the experiment.

Ingested Data: we decided to add 2 columns which indicated the city and the year of the measurement

We aimed to visualize the data using Tableau, which is one of the most commonly used Business Intelligence (BI) tools. Hence, we needed to transform the data in a format that is easily and efficiently consumed by Tableau. It is worth mentioning that we scraped the data in a format that was already structured, and, therefore, very little data cleaning was required. We merely reordered and reformatted some columns and checked that there were no null values.