The Code: On Data Exploration and Visualisation
By: Sam Taylor-Offord
Tags: Data Visualization, Programming, Data Science, Matplotlib, Python

The Code: On Data Exploration and Visualisation

In my last post I took us on a journey through the data exploration and visualisation process. I kept things pretty conceptual there; I didn’t want to scare off anyone who wasn’t a coding aficionado… but I know such people are out there (perhaps reading this right now?) and the last thing I want to do is tease them with nice plots and then give no code!

Now, fair warning: this post will get more complicated as it goes on. That’s just the nature of the beast: the prettier the plot, the meaner its backside. I’ll do what I can to keep you following and to keep you entertained, but I won’t cry if you tag out (although I will smile if you persevere, and perhaps one day you’ll thank yourself for it, who knows?).

As I’m sure many of you will know, there is a lot — and I mean a lot — of work that goes into getting data ready for plotting. That said, plotting the data is often the first thing we want to do, even if the data is far from its final form. It’s never too early to plot your data after all!

# First, import the module we need to do the plotting import matplotlib.pyplot as plt

And there’s the first step. What a line! Every Pythonista out there will be familiar with this bad boy, and every wannabe programmer should feast their eyes on the king of plotting: matplotlib. Now before I say anything more I want to address those among you who are surely thinking, “Python? This guy uses Python? Doesn’t he know Python isn’t a real programming language?” and say, well, yes, Python is perhaps not as elegant as other programming languages, but let’s not gatekeep programming; any programming language is valid if it meets our needs, and Python has the benefit of being easy to read and understand in the form of, say, a blog post. Anyway, here’s the first plot we’ll make:

Figure 1: scatter plot of magnitudes values of mww magnitude type and mB magnitude type. Data are shown by blue dots. The dotted line is that with slope 1 and intercept 0 and is plotted for reference. The solid line is that fit to the data by orthogonal regression whose parameters are presented in the plot title.

And the code to make it:

# First we need to extract data from the object its spooled up in so it can be plotted. # Here I use the datalist variable to hold the data. This is a list of lists, with each list containing a column of data, and all columns containing the same number of data. A corresponding data_types list exists which contains the column headers as strings. Before we can plot any data in this list, we need to extract it. n, m = 0, 1 # Set which columns to extract data from

x = datalist[n] # x is the nth column of data

y = datalist[m] # y is the mth column of data # Now we have the data to plot as x and y. Let's assume every entry in these lists are of a plottable type, i.e. int or float, and plot it: plt.scatter(x, y, s=1) # Scatterplot the data with point size 1 # Simple. We now have a scatter plot of x and y values. To add context to the plots we'll add axis labels, a grid, a line of slope 1 and intercept 0, and then we'll set the axis limits to a range we know will contain all the data without being too "zoomed out". plt.xlabel(data_types[n])

plt.ylabel(data_types[m])

plt.grid(which='major', axis='both', alpha=0.5)

plt.plot(range(11), range(11), color='k', linestyle='--', linewidth=0.5, alpha=0.5)

plt.xlim(2, 10)

plt.ylim(2, 10) # Lastly, we'll ensure the plot is square by forcing an equal aspect plt.gca().set_aspect('equal', adjustable='box') # And of course, show the plot! plt.show()

Phew, that’s a lot of commenting! I hope that’s enough for you, or that it’s not too much; I’m always trying to strike the balance between ensuring a code’s well documented without having comments crowd the code. If you feel there’s more needing to be said I recommend checking out the documentation for Matplotlib. While it’s a big module with a big learning curve, the payoff is real. I recommend the struggle.

Now I’m going to move on, and you might say “Hey, are you going to move on without talking about that other line, and the title, are you?” Well, yes, I was hoping to. As much as I love line plotting, there’s only so much content I want to put in any given blog post! I promise that I’ll go into great detail on line plotting in the future, but not today. Soon you’ll see why.

Figure 2: scatter plot of magnitudes values of mB magnitude type and Mw magnitude type. Data are shown as blue dots. The dotted line is that with slope 1 and intercept 0 and is plotted for reference. Mean values of the mB data in 0.1 unit Mw bins are plotted in orange and scaled corresponding to the number of data points in each bin. The total number of data used in the plot is shown in the title.

Alright, Figure 2. The basics are pretty much the same, but we’ll do a little data manipulation ahead of the plotting to get those average values and their sizes:

import numpy as np # Import NumPy for the data processing # Get data for plotting: use a loop to check each data point is not a NaN value, as we do not want NaN values corrupting the data manipulation. m, n = 0, -1

x, xdec = [], []

y = []

for k in range(len(datalist[0])):

if not np.isnan(datalist[n][k]) and not np.isnan(datalist[m][k]):

x.append(datalist[m][k])

xdec.append(round(datalist[m][k]), 1)

y.append(datalist[n][k]) # Here we have produced the familiar lists x and y, but also xdec, which is the decimated (rounded to nearest 0.1) x value. We'll need this for the averaging. # Plot the data, as before plt.scatter(x, y, s=1) # Bin the y data to 0.1 unit x data bins. x_vals = list(set(xdec)) # Produce the list of unique xdec values

binned_yvals = [[] for l in range(len(x_vals))]

for l in range(len(xdec)):

x_idx = x_vals.index(xdec[l])

binned_yvals[x_idx].append(y[l]) # Calculate mean y values in each bin and the number of data points in each bin. mean_yvals = [0] * len(binned_yvals)

n_yvals = [0] * len(binned_yvals)

for l in range(len(binned_yvals)):

mean_yvals[l] = np.mean(binned_yvals[l])

n_yvals[l] = len(binned_yvals[l])

n_tot = sum(n_yvals) # Plot the average y value for each bin, scaled to the number of data in that bin. plt.scatter(x_vals, mean_yvals, s=n_yvals, edgecolors='k', linewidths=1)



# Plot a line showing the 1:1 relationship for reference



plt.plot(range(11), range(11), color='k', linestyle='--', linewidth=0.5, alpha=0.5)



# Add plot features for clarity



plt.xlabel(data_types[m])

plt.ylabel(data_types[n])

plt.grid(which='major', axis='both', linestyle='-', alpha=0.5)

plt.xlim(2, 9)

plt.ylim(2, 9)

plt.gca().set_aspect('equal', adjustable='box')

plt.title('N=' + str(n_tot)) # Show the plot plt.show()

So there’s a bit more to unpack here. The basics are the same: extract some data as lists, plot each entry of one list against the respective entry of the other, then show the plot (with some adornments). The real addition is the data averaging. Here, the x values are first rounded to the nearest 0.1, then a mean of the y values in each 0.1 unit of x value is taken. You can think of this process as splitting the x axis up into 0.1 unit windows, then grouping in each window all the y values falling in that window before averaging all those y values. You’ll want to have this idea sorted before we hit the next level, which we’ll do right now.

Figure 3: scatter plot of magnitudes values of mB magnitude type and Mw magnitude type. Data are shown as filled dots with the number of data at each point denoted by dot colour. Data points are rounded to the nearest 0.1 prior to plotting to allow the data distribution to be shown. The dotted line is that with slope 1 and intercept 0 and is plotted for reference. Mean values of the mB data in 0.1 unit Mw bins are plotted as open circles. The normalised distribution of each dataset is shown on the respective axis. The total number of data used in the plot is shown in the title.

Damn that’s a sexy plot. The problem with sexy plots? A lot of work goes into making them. I apologise in advance, but this is going to get complicated. That said, we’ve already covered a lot of the ideas, so bear with me, and trust me that I’ll walk you through it all! First, let’s flick over all the code before we break it down after:

import numpy as np # Bin the y data to the corresponding 0.1 unit bin of x data and calculate the data distribution and number of points in each bin m, n = 0, -1

x, xdec = [], []

y, ydec = [], []

for k in range(len(datalist[0])):

if not np.isnan(datalist[n][k]) and not np.isnan(datalist[m][k]):

x.append(datalist[m][k])

xdec.append(round(datalist[m][k]), 1))

y.append(datalist[n][k])

ydec.append(round(datalist[m][k]), 1)



x_vals = list(set(xdec))

y_vals = list(set(ydec))

binned_y = [[] for l in range(len(x_vals))]

binned_ydec = [[] for l in range(len(x_vals))]

binned_yvals = [[] for l in range(len(x_vals))]

binned_ycounts = [[] for l in range(len(x_vals))]

bin_means = [0] * len(x_vals)

for l in range(len(xdec)):

x_idx = x_vals.index(xdec[l])

binned_y[x_idx].append(y[l])

binned_ydec[x_idx].append(round(y[l], 1))

for l in range(len(x_vals)):

binned_yvals[l] = list(set(binned_ydec[l]))

for k in range(len(binned_yvals[l])):

binned_ycounts[l].append(binned_ydec[l].count(binned_yvals[l][k]))

bin_means[l] = np.mean(binned_y[l])

n_tot = len(y)



# Normalise y count values for plotting maxcount = 1

for l in range(len(binned_ycounts)):

for k in range(len(binned_ycounts[l])):

if binned_ycounts[l][k] > maxcount:

maxcount = binned_ycounts[l][k]

for l in range(len(binned_ycounts)):

for k in range(len(binned_ycounts[l])):

binned_ycounts[l][k] /= maxcount



# Plot data cm = plt.cm.get_cmap('RdYlBu_r')

# ^ Get the reversed Red-Yellow-Blue colourmap for l in range(len(x_vals)): # Prepare the list of colours for plotting colors = []

for k in range(len(binned_ycounts[l])):

colors.append(cm(binned_ycounts[l][k])) # Perform the plotting plt.scatter([x_vals[l]] * len(binned_yvals[l]), binned_yvals[l], c=colors, s=10, cmap=cm) # When the x bin containing the highest data point count occurs, add the colourbar to the figure if 1 in binned_ycounts[l]:

plt.set_cmap('RdYlBu_r')

cb = plt.colorbar()

cb.set_ticks(cb.get_ticks())

cb.set_ticklabels(ticklabels=[int(round(tickvalue * maxcount)) for tickvalue in cb.get_ticks()],

update_ticks=True)

cb.set_label('data count', labelpad=15, rotation=270)



# Plot data means plt.scatter(x_vals, bin_means, s=20, ec='black', fc='None', lw=1)



# Plot a line showing the 1:1 magnitude relationship for reference plt.plot(range(11), range(11), color='k', linestyle='--', linewidth=1, alpha=0.8)



# Plot each data's distribution along the axes x_distro = [0] * len(x_vals)

y_distro = [0] * len(y_vals)

for idx, x_val in enumerate(x_vals):

x_distro[idx] = xdec.count(x_val)

for idx, y_val in enumerate(y_vals):

y_distro[idx] = ydec.count(y_val) if len(x_distro) > 0:

max_xdistro = max(x_distro)

for idx in range(len(x_distro)):

x_distro[idx] = x_distro[idx] / max_xdistro + 2

x_vals, x_distro = zip(*sorted(zip(x_vals, x_distro)))

plt.plot(x_vals, x_distro, color='k', linewidth=1)



if len(y_distro) > 0:

max_ydistro = max(y_distro)

for idx in range(len(y_distro)):

y_distro[idx] = y_distro[idx] / max_ydistro + 2

y_vals, y_distro = zip(*sorted(zip(y_vals, y_distro)))

plt.plot(y_distro, y_vals, color='k', linewidth=1)



# Add plot features for clarity plt.xlabel(data_types[m])

plt.ylabel(data_types[n])

plt.grid(which='major', axis='both', linestyle='-', alpha=0.5)

plt.xlim(2, 9)

plt.ylim(2, 9)

plt.gca().set_aspect('equal', adjustable='box')

plt.title('N=' + str(n_tot))

Hot damn! What was that!?

Yeah, I feel the same way, and I wrote it!

Banter aside, let’s unpack this monster of a code cell.

m, n = 0, -1

x, xdec = [], []

y, ydec = [], []

for k in range(len(datalist[0])):

if not np.isnan(datalist[n][k]) and not np.isnan(datalist[m][k]):

x.append(datalist[m][k])

xdec.append(round(datalist[m][k]), 1))

y.append(datalist[n][k])

ydec.append(round(datalist[m][k]), 1)

This bits pretty easy: we go through all the data points, check that they’re not NaN values, then put them and their rounded-to-the-nearest-0.1-versions into some lists. The main difference here is that rounded versions of both the x and y values are calculated.

x_vals = list(set(xdec))

y_vals = list(set(ydec))

binned_y = [[] for l in range(len(x_vals))]

binned_ydec = [[] for l in range(len(x_vals))]

binned_yvals = [[] for l in range(len(x_vals))]

binned_ycounts = [[] for l in range(len(x_vals))]

bin_means = [0] * len(x_vals)

for l in range(len(xdec)):

x_idx = x_vals.index(xdec[l])

binned_y[x_idx].append(y[l])

binned_ydec[x_idx].append(round(y[l], 1))

for l in range(len(x_vals)):

binned_yvals[l] = list(set(binned_ydec[l]))

for k in range(len(binned_yvals[l])):

binned_ycounts[l].append(binned_ydec[l].count(binned_yvals[l][k]))

bin_means[l] = np.mean(binned_y[l])

n_tot = len(y)

Ok, this bit will give us some pause. The first few lines are just variable setup, so not too much to look into here. In the loops we find the meat of it: the first loop bins y values and their rounded equivalents to the x bins, and the second loop looks in each x bin and finds the y bins in that bin in preparation for the nested loop. We could just take all possible y bins (all 0.1 values between the minimum and maximum y value), but that will introduce 0 values to the data where no data exists, and that’s a mess I’d rather not deal with. Anyway, we need to look at the nested loop before you’ll know what I’m on about:

for k in range(...

In this loop the code works through every y bin in the current x bin and counts the number of data points that fall into both those bins. Think of this process as targeting those data within each square of 0.1 unit width, then counting how many data there are in that square.

Some of you may think, “Hmm, that kind of sounds like a grid, is he gridding the data?” Well, yes; through the binning process we have effectively defined a grid of points at 0.1 unit intervals over both the x and y dimensions, and the square I was talking about is the “sampling region” of each point: 0.05 units from the point in both directions. The code looks through each column of the grid (the x bin) and sees many data points are in each row of the grid (the y bin). We’re assuming here that there will be more than 1 data point in each sampling region. In Figure 2 we saw that there were a lot of data points, but that mean values did not fall in the middle of each x bin’s y-scatter. This tells us that there’s some overlap of the data going on. Whether that overlap is some data point on data point action, or if data points are only slightly overlapping doesn’t matter in this method: we just say that if you’re a data point and you fall in the sampling region then you’re getting assigned to the grid point and that’s that.

So that’s what’s going on here. As well as counting the data points at each grid point the code also gets the mean of the y data in the x bin.

# Normalise y count values for plotting maxcount = 1

for l in range(len(binned_ycounts)):

for k in range(len(binned_ycounts[l])):

if binned_ycounts[l][k] > maxcount:

maxcount = binned_ycounts[l][k]

for l in range(len(binned_ycounts)):

for k in range(len(binned_ycounts[l])):

binned_ycounts[l][k] /= maxcount # Plot data cm = plt.cm.get_cmap('RdYlBu_r')

# ^ Get the reversed Red-Yellow-Blue colourmap colorbar = False

for l in range(len(x_vals)): # Prepare the list of colours for plotting colors = []

for k in range(len(binned_ycounts[l])):

colors.append(cm(binned_ycounts[l][k])) # Perform the plotting plt.scatter([x_vals[l]] * len(binned_yvals[l]), binned_yvals[l], c=colors, s=10, cmap=cm) # When the x bin containing the highest data point count occurs, add the colourbar to the figure if 1 in binned_ycounts[l] and colorbar is False:

colorbar = True

plt.set_cmap('RdYlBu_r')

cb = plt.colorbar()

cb.set_ticks(cb.get_ticks())

cb.set_ticklabels(ticklabels=[int(round(tickvalue * maxcount)) for tickvalue in cb.get_ticks()],

update_ticks=True)

cb.set_label('data count', labelpad=15, rotation=270)

Plotting time. This section starts by normalising the grid point data counts ( binned_ycounts ) to the highest value, a move that will aid us when plotting.

Next we setup a colourmap for the plotting. I chose to use a colourmap that would give good contrast between grid point data counts while being quite intuitive: red is lots, blue is not, yellow is middling.

The plotting then begins with the code going through each x bin and plotting the grid point data counts ( binned_ycounts ) that fall in that bin. The data count is indicated by the point colour, which is taken from the colourmap. Here the normalised nature of the counts is helpful as we can get colours from the colourmap using floats between 0 and 1 (our normalised count values!). A nice trick.

Eventually the code will hit the x bin that contains the highest grid point data count. This x bin will have data extending to the highest value on the colourbar, and we can use this to add the colourbar to the plot with the full range of colours shown (a surprisingly difficult task). As the data itself is normalised, we need to change the colourbar tick labels to show the true values of the data.

# Plot each data's distribution along the axes x_distro = [0] * len(x_vals)

y_distro = [0] * len(y_vals)

for idx, x_val in enumerate(x_vals):

x_distro[idx] = xdec.count(x_val)

for idx, y_val in enumerate(y_vals):

y_distro[idx] = ydec.count(y_val) if len(x_distro) > 0:

max_xdistro = max(x_distro)

for idx in range(len(x_distro)):

x_distro[idx] = x_distro[idx] / max_xdistro + 2

x_vals, x_distro = zip(*sorted(zip(x_vals, x_distro)))

plt.plot(x_vals, x_distro, color='k', linewidth=1)



if len(y_distro) > 0:

max_ydistro = max(y_distro)

for idx in range(len(y_distro)):

y_distro[idx] = y_distro[idx] / max_ydistro + 2

y_vals, y_distro = zip(*sorted(zip(y_vals, y_distro)))

plt.plot(y_distro, y_vals, color='k', linewidth=1)

Here’s the last truly new bit. This section starts by taking all the grid point data counts and summing them along each bin along each axis. Unlike what we saw before, this is a relatively simple operation! This actually produces the distribution of each dataset at 0.1 unit resolution. To include these distributions on the plot the code normalises the data count values then adds 2 as this is the lower limit of each axis. This is another trick: instead of wrangling multiple axes on the same plot (short version: a nightmare) I just adjust the values of the second and third datasets (the distributions) so they plot where I want them to but on the existing axis. Ha, take that matplotlib! (Not shown: the many hours I spent trying to figure this out.)