Visualising temperatures in Amsterdam as a heatmap in R — Part I
By: Deepak Gulati
Tags: Heatmap, Data Science, R, Ggplot2

I was recently playing with heatmaps and decided to use one to visualise temperature data for Amsterdam.

Obtaining the data:

The Royal Dutch Meteorological Institute (Koninklijk Nedarlands Meteorologisch Instituut in Dutch, or KNMI for short) provides historical weather data going back to 1950s for free. You can get the data for a weather station of your choice from their measurements and observations page (in Dutch). They give both hourly and daily readings. I got hold of the daily data for Schiphol (weather station no. 240) from this page.

Reading and understanding the data:

The data is formatted as a CSV file. The first few lines have explanations in Dutch and English describing each field. These lines have not been prefixed with any comment character (typically #), so we’ll skip them when we read the file. The content we are interested in begins at row 48 — the header row.

t <- read.csv('~/Downloads/etmgeg_240.txt', skip=47)

str(t) 'data.frame': 24311 obs. of 41 variables:

$ X..STN : int 240 240 240 240 240 240 240 240 240 240 ...

$ YYYYMMDD: int 19510101 19510102 19510103 19510104 19510105 19510106 19510107 19510108 19510109 19510110 ...

$ DDVEC : int 188 153 203 193 207 185 240 203 246 176 ...

$ FHVEC : int 77 41 15 77 82 57 134 77 67 51 ...

$ FG : int 87 41 21 77 87 62 144 82 72 62 ...

$ FHX : int 195 82 51 103 144 123 180 103 129 103 ...

$ FHXH : int 18 4 24 15 21 22 13 20 12 21 ...

$ FHN : int 41 10 0 51 36 0 62 51 21 26 ...

$ FHNH : int 24 21 3 1 24 3 24 1 24 3 ...

$ FXX : int NA NA NA NA NA NA NA NA NA NA ...

$ FXXH : int NA NA NA NA NA NA NA NA NA NA ...

$ TG : int 12 13 3 12 48 64 67 61 41 26 ...

$ TN : int -13 7 -20 -7 18 46 58 36 6 -8 ...

$ TNH : int 1 4 9 6 1 4 24 2 24 5 ...

$ TX : int 26 18 16 19 84 86 86 70 63 61 ...

$ TXH : int 20 19 23 16 21 24 1 14 1 24 ...

$ T10N : int NA NA NA NA NA NA NA NA NA NA ...

$ T10NH : int NA NA NA NA NA NA NA NA NA NA ...

$ SQ : int NA NA NA NA NA NA NA NA NA NA ...

$ SP : int NA NA NA NA NA NA NA NA NA NA ...

$ Q : int NA NA NA NA NA NA NA NA NA NA ...

$ DR : int NA NA NA NA NA NA NA NA NA NA ...

$ RH : int NA NA NA NA NA NA NA NA NA NA ...

$ RHX : int NA NA NA NA NA NA NA NA NA NA ...

$ RHXH : int NA NA NA NA NA NA NA NA NA NA ...

$ PG : int 9891 9876 10019 10098 10059 10043 10026 10084 10089 10090 ...

$ PX : int 9957 9923 10097 10114 10094 10087 10105 10115 10151 10148 ...

$ PXH : int NA NA NA NA NA NA NA NA NA NA ...

$ PN : int 9837 9853 9931 10084 10030 9980 9957 10040 10034 10016 ...

$ PNH : int NA NA NA NA NA NA NA NA NA NA ...

$ VVN : int NA NA NA NA NA NA NA NA NA NA ...

$ VVNH : int NA NA NA NA NA NA NA NA NA NA ...

$ VVX : int NA NA NA NA NA NA NA NA NA NA ...

$ VVXH : int NA NA NA NA NA NA NA NA NA NA ...

$ NG : int 7 8 6 7 8 8 8 7 5 5 ...

$ UG : int 90 93 94 94 95 89 82 93 89 89 ...

$ UX : int 98 98 100 97 100 100 94 97 96 95 ...

$ UXH : int 6 9 21 12 5 3 4 8 1 2 ...

$ UN : int 73 88 83 89 89 72 76 88 76 78 ...

$ UNH : int 20 1 12 8 14 23 16 17 14 12 ...

$ EV24 : int NA NA NA NA NA NA NA NA NA NA ...

So we have a wide variety of weather observations — although readings for some metrics are not available for every year. The farther you go in the past, the sketchier the data. For our exercise, let’s get started with just the daily temperature data for the year 2016. YYYYMMDD is the date colum, while TX and TN columns contain the maximum and minimum temperatures respectively. The temperatures are in 0.1ºC units (e.g. 10ºC appears as 100 in this file), so we’ll divide them by 10.

Filtering and modifying the data:

We’ll use the filter function from the dplyr package to filter the data for 2016. We’ll then use the select function to pick just the fields that we need and assign this subset to a new dataframe.

library(dplyr)

t %>% filter(YYYYMMDD >= 20160101 & YYYYMMDD <= 20161231) %>% select(YYYYMMDD, TN, TX) -> t.2016

Next, we’ll make some changes to this data. The date for each observation (the YYYYMMDD column) is stored as an integer. We’ll convert it to a proper date object and then break it into handy units — year, month, week, weekday and day — for later use. While we are at it, we’ll also divide the temperatures by 10. We’ll use the lubridate package for dealing with the dates.

library(lubridate)

t.2016 %>% mutate(

date = as.Date(as.character(YYYYMMDD), format="%Y%m%d"),

year = year(date),

month = month(date, label=T, abbr=T),

week = strftime(date,"%W"),

wday = substring(wday(date, label=T, abbr=T),1,2),

day = day(date),

TN = TN / 10,

TX = TX / 10

) -> t.2016

head(t.2016) YYYYMMDD TN TX date year month week wday day

1 20160101 1.1 8.0 2016-01-01 2016 Jan 00 Fr 1

2 20160102 4.6 6.5 2016-01-02 2016 Jan 00 Sa 2

3 20160103 5.3 8.5 2016-01-03 2016 Jan 00 Su 3

4 20160104 4.3 7.2 2016-01-04 2016 Jan 01 Mo 4

5 20160105 4.4 7.7 2016-01-05 2016 Jan 01 Tu 5

6 20160106 1.8 4.4 2016-01-06 2016 Jan 01 We 6

Visualising the data:

Let’s take a first stab a visualising the maximum temperatures as a heatmap. We’ll use the viridis colour palette for our heatmap.

library(ggplot2)

library(viridis)

ggplot(data = t.2016, aes(x=day,y=month)) + geom_tile(aes(fill=TX)) + scale_fill_viridis()

We can clearly see the spread of summer in Amsterdam from May-Sep, the warmest day of the year in July and patches of very cold days in Jan and Dec.

We’ll now make a few more minor tweaks:

We’ll label all the days on x-axis rather than have ggplot pick something for us. After all, we know that a month doesn’t have more than 31 days. While we are at it, we’ll make sure that the days (x-axis) start from 1 and not from 0. We’ll also nudge the y-axis a little closer to the heatmap. Since these are maximum temperatures, I’d like to pick a colour palette that evokes warmth. Viridis comes with 3 more colour palettes — magma, plasma and inferno and we’ll pick the magma theme to do the job. In the screenshot above, the tiles look pretty nice but depending on the size of your plot window, they might come out looking squished. We’ll lock the tiles to always be in a 1:1 ratio — irrespective of the width and height of the plot window. As a final touch, we’ll hide the grey grid in the background by using a different theme. I am fan of the tufte theme from the ggthemes package.

library(ggthemes)

ggplot(data = t.2016, aes(x = day,y = month)) +

geom_tile(aes(fill = TX)) +

scale_x_continuous(breaks=c(1:31), expand=c(0,0)) + coord_equal(ratio = 1) +

scale_fill_viridis(option="magma") + theme_tufte(base_family="Helvetica")

I personally find that the summer months stand out even better with the magma theme.

Links:

If you like the viridis library, do read the vignette. I also found the SciPy2015 video mentioned in the vignette very educational. I learned a lot of the techniques I used here from this post by Bob Rudis.

Up Next:

While I am quite pleased with how far we got here, there is a lot more that we can do with a year worth of temperature data. In the next post, I will share an approach to make the heatmap look like an actual calendar.

Update: Part II is now up as is the source code for both parts.