Colourscales are everything in data visualisation – are you using them effectively?
By: Jp Hwang
Tags: Programming, Python, Data Science, Data Visualization, Technology

Colourscales are everything in data visualisation – are you using them effectively?

How choosing the right colour and scales can make or break your message (with examples & Python code)

You might have seen this COVID map — first reported by andishenouraee on Twitter, and then covered by the media at large.

Even if you have, take another look. If you haven’t, these are choropleth maps showing Covid-19 cases per county (from July 2020) in the state of Georgia. The two maps appear very similar, but the observant among you might have noticed one crucial difference — the scale.

Changes to the scale mask the fact that each colour band is about 50% higher, and many of the similarly-coloured areas would in fact, under the same scale, be a shade darker on the right-hand image.

This approach is fine if the goal is to simply present relative data – i.e. highlight the county with the most or least cases. But it’s not so great for much else.

Oh, and also — those scales look odd! The numbers defining these ranges are quite irregular. Let’s just visualise the data. We can set the colour bands to the same colours as the map to make a stacked bar chart:

bands_df = pd.read_csv("srcdata/colormap_1.csv")

fig = px.bar(bands_df, x="Width", y="Type", color="Band", barmode="stack", template="plotly_white",

color_discrete_sequence=["#e9f3fa", "#9cbfe9", "#74a6e1", "#4182d6", "#2d5a95", "#c8281c"])

fig.update_layout(bargap=0.4)

fig.show()

The map scales — visualised (Image: Author)

The bottom band shows the scale in the “old” map, and the top band the “new” map with the more recent data.

Whatever your thoughts are about the validity of the initial criticism, it is difficult to deny that the colours are confusing and arbitrary, leading to an unclear message. But you wouldn’t have known any of that from just the output.

So, as data people — data scientists, analysts, visualisation experts — we must be careful to make sure that the way the information is being presented is consistent with the goal. That is, the visualisation should be aligned with the message.

Conversely, each data visualisation should be looked at with your BS detector switched on and aware that the message might be that of misinformation.

Let’s take a look at some other examples of how colourscales can change the message delivered by the visualisation.

Here, I will use a few modified examples of my map data from this below article. If you would like to follow along, the original article presents the dataset and code in the linked git repo, and the new code is contained in colourscale.py . I will show revised code where it might be appropriate.

Stuck in the middle

Importance of midpoints in diverging scales

I probably don’t need to say that automatic scaling is usually not what we are after. Here’s the map with automatic scaling.

Example 0: With an automatic colorscale (Image: Author)

It shows some contours, but the map is dominated by the orange/red colours, although the entire colourscale spans transitions from blue/yellow/red. Here’s another approach.

Example 1: Colorscale median at median of counties (Image: Author)

This map is plotted with a range between 0 and US$100,000, based on the fact that the median value of counties’ typical incomes are about US$50,000. The end result is a map that divides the United States’ counties to two halves, showing in blue 50% of counties with the highest incomes, and in red the other half of counties.

Compare that to the map below where the midpoint of the colourscale is now the overall U.S. median household income.

Example 2: Colorscale median at national median (Image: Author)

While it is a similar graph overall, changing the midpoint to the national median means the colours significantly, showing a lot more red / orange than before.

What is it showing?

This figure compares counties’ typical incomes with the entire country’s typical income. In other words, the counties in red / orange are those whose typical income is below the national median. This graph clearly shows that most of the country, at the county level, has a median well below the national median.

It’s a simple change, but a significant one.

Let’s take a look at a few more substantial changes.

What if we just wanted to know which counties are the wealthiest, and which the poorest?

Relative scales & colours

We need to rearrange the data first of all. The .rank() method can be used here to generate a rank of the data from a column, such as income, like so:

data_df["Median_Household_Income_2018"].rank())

This will generate a rank where the highest income counties will have the highest “rank”. For intuitiveness, let’s invert that, and assign it to a new column:

data_df = data_df.assign(inc_rank=data_df["Median_Household_Income_2018"].rank(ascending=False))

And so, we can now plot the data:

fig = px.choropleth_mapbox(data_df, locations="fips", color="inc_rank", geojson=counties, color_continuous_scale=px.colors.diverging.RdYlBu, labels=labels)

Example 3: Ranking of counties, using a diverging scale (Image: Author)

But is a divering scale that good a thing to do? A diverging scale tends to wash out the middle of the scale, while really visually accentuating the ends of the scale. The midpoint of our rankings and the difference from it don’t mean anything, so changing from red to blue doesn’t really denote anything.

Instead, let’s use a sequential colorscale.

fig = px.choropleth_mapbox(data_df, locations="fips", color="inc_rank", geojson=counties, color_continuous_scale=px.colors.sequential.Greens_r, labels=labels)

Resulting in:

Example 4: Ranking of counties, using a sequential scale (Image: Author)

Better! It’s cleaner, you’re not distracted by the meaningless middle, or the changes to the hue for no reason.

What would this chart look like with some other sequential colour scales? Take a look at these examples, with different colourscales:

fig = px.choropleth_mapbox(data_df, locations="fips", color="inc_rank", geojson=counties, color_continuous_scale=COLORSCALE_HERE, labels=labels)

Example 4.1: In Blues (Image: Author)

Example 4.2: In Reds (Image: Author)

Example 4.3: In Viridis (Image: Author)

Example 4.4: In Sunsetdark (Image: Author)

Example 4.5: In Green, reversed (Image: Author)

They each change the tone of the image and the accompanying message, even when the data is exactly the same. What is the impact of using a “cold” colour like blue, or a “warm” colour like red for the shading? Red, for one, is more of an evocative colour. Is that what you want?

Do you want to choose a darker colour to indicate lower socioeconomic regions? What about using red or blue for those? Might the association evoke an emotional reaction (intended or not) and detract from the message?

The colour palette is a huge topic and one I’m not even remotely qualified to talk about — but make sure to explore different options, and how they “feel”.

Putting the scale in a colourscale

Let’s stop and talk about what information we seek to present. We’ve been ranking counties as is, but they’re a) not equally sized, and b) have huge differences in population densities. Take a look at this chart showing the disparities in population sizes in counties.

fig = px.histogram(data_df, x="Median_Household_Income_2018", nbins=20, labels=labels, color_discrete_sequence=px.colors.qualitative.Safe,

template="plotly_white", title="Histogram of counties' populations")

fig.show()

Counties’ populations are spread over a wide range (Image: Author)

It stretches all the way to cities with 10million+ population, whereas the majority of the population lives in counties under 200k. What if we look at the data in a log scale? Plotly allows plotting in log scale, but we can also generate a new column where it’s been converted into log scale:

fig = px.histogram(data_df, x="pop_log", nbins=20, labels=labels,

color_discrete_sequence=px.colors.qualitative.Safe, template="plotly_white", title="Histogram of counties' populations")

fig.show()

A log range works much better to describe this distribution! (Image: Author)

And going another step further, splitting the dataset by income reveals that the lower income set includes far more lower-income datasets.

Far more smaller counties are in the lower-income subset (Image: Author)

Based on all this, the colourscale should perhaps be scaled by the not just the number of counties, but its population. If you need further proof, look at this scatter plot showing the distribution of county incomes and income:

fig = px.scatter(data_df, x="inc_rank", y="Median_Household_Income_2018", labels=labels,

size="POP_ESTIMATE_2018", template="plotly_white")

fig.show()

Bubble chart of all US counties — the large counties generally have higher incomes (Image: Author)

So let’s scale our colour map to take account of the county population. The way I did it was to determine ranks, and then to re-adjust the percentile value by the population sizes. The code snippet is here:

data_df = data_df.assign(inc_percentile=data_df["Median_Household_Income_2018"].rank(pct=True, ascending=False) * 100)

total_pop = data_df["POP_ESTIMATE_2018"].sum()

data_df = data_df.assign(pop_inc_percentile=data_df["inc_percentile"].apply(lambda x: data_df[data_df["inc_percentile"] <= x]["POP_ESTIMATE_2018"].sum()/total_pop * 100))

The resulting graph (in the same manner as before) is drawn as such:

fig = px.choropleth_mapbox(data_df, locations="fips", color="pop_inc_percentile", geojson=counties, color_continuous_scale=px.colors.sequential.Greens_r, labels=labels)

Example 5: Ranking of counties, adjusted by population (Image: Author)

Of course, the data is similar and therefore the resulting visualisation will be, too. But the difference is extremely stark. Take a look for yourself.

A comparison of visualisations — counties’ rankings vs population-adjusted rankings

Interesting, isn’t it?