How to create fast and accurate scatter plots with lots of data in python | by Paul Gavrikov | Towards Data Science

Scatter plots are quite basic and easy to create — or so I thought. Recently I had to visualize a dataset with hundreds of millions of data points. If you’re a Python developer you’ll immediately import matplotlib and get started. But it turns out there are better, faster, and more intuitive ways to create scatter plots.
What’s the problem with matplotlib? Well, matplotlib is a great Python library and is definitely part of the data science must-knows. But matplotlib is also a huge all-rounder and may perform suboptimally in some scenarios. This is one of those.
Let’s say we have an array X and its shape is (1_000_000, 2). Each column represents one axis. So, any row is a coordinate.
import matplotlib.pylot as pltplt.scatter(X[:, 0], X[:, 1])plt.show()
So that produces a scatter plot but we have no idea if points overlap or generally about the intensity of a region. We can fix this by setting the s and alpha parameters. The first parameter controls the size of each point, the latter gives it opacity. We’d like size and opacity that allows us to distinguish between different points. And this is where trial and error begins. And as your data size increases, this process gets more and more painful.
import matplotlib.pylot as pltplt.scatter(X[:, 0], X[:, 1], s=1, alpha=0.1)plt.show()
datashaderis a great library to visualize larger datasets. The main improvement comes from the rasterization process: matplotlib will create a circle for every data point and then, when you’re displaying your data, it will have to figure out which pixels on your canvas each point occupies. Typically, each point will occupy multiple pixels. Instead, datashader will divide your 2D-space into width horizontal and height vertical bins. And then it just checks which bin each sample occupies. If this process sounds familiar to you, then that’s because that’s how you create a histogram. In this case, a 2D-histogram with equal-width bins. The histogram you’ve created is already the same shape as your image. So all that’s left is to apply the colormap. I really like fire from the colorcet library. And you’re done. Note that datashader only accepts DataFrame as input (be it pandas , dask or others) and your data must be stored as float32.
There are the additional packages we need:
pip install datashader, colorcet, pandas
And that is the code:
import datashader as dsimport pandas as pdimport colorcet as ccimport matplotlib.pyplot as pltdf = pd.DataFrame(data=X, columns=["x", "y"])  # create a DF from arraycvs = ds.Canvas(plot_width=500, plot_height=500)  # auto range or provide the `bounds` argumentagg = cvs.points(df, 'x', 'y')  # this is the histogramimg = ds.tf.set_background(ds.tf.shade(agg, how="log", cmap=cc.fire), "black").to_pil()  # create a rasterized imageplt.imshow(img)plt.axis('off')plt.show()
Dots can no longer partially overlap, and since you’re creating a histogram the colormap will handle your previous opacity problem. You can decide how you want to normalize the data (using the how parameter). Here we set it to be logarithmic, but typically I’d recommend leaving it at the default which uses a more sophisticated method.
And now let’s just add a color bar to the plot. Ugh! How? All we have now is the histogram and a rasterized image. We can display the image with matplotlib but have no information of the colormap. There’s probably some hack, but let’s be honest: It would be nothing more than a dirty hack and could introduce a lot of confusion. So, datashader is great, fast, and easy to use — but it comes at a price: no color bars and no interactive plots (i.e. no GUI which allows to zoom, rotate, etc.). It really shines at creating external graphics, though. So, give it a try! Especially, when you’re dealing with geolocation data.
Dynamic plots aren’t that important to me, but I really needed color bars. So, I went ahead and coded up my own solution. It’s actually really simple. Initially, I was using numpy to compute a 2D-Histogram and then let matplotlib handle the shading. But numpy.histogram2d is quite slow, which is why I switched to fast_histogram. I am again using the colorcet.fire map but accessing it via the cc.cm dict to be compatible with matplotlib . Also, I am supplying the norm argument to use a logarithmic colormap. Be aware that vmin=0 is invalid because the logarithm of zero is not defined. This is also why all 0 values are mapped to what’s called the bad color. So just set the bad color to the color for the smallest value (or to whatever color you want your background to be).
There are the additional packages we need:
pip install fast-histogram colorcet
And run the code:
import colorcet as ccimport matplotlib.colors as colorsimport matplotlib.pyplot as pltfrom fast_histogram import histogram2dcmap = cc.cm["fire"].copy()cmap.set_bad(cmap.get_under())  # set the color for 0bounds = [[X[:, 0].min(), X[:, 0].max()], [X[:, 1].min(), X[:, 1].max()]]h = histogram2d(X[:, 0], X[:, 1], range=bounds, bins=500)plt.imshow(h, norm=colors.LogNorm(vmin=1, vmax=h.max()), cmap=cmap)plt.axis('off')plt.show()
Note that the created scatter plots are rotated, due to the way how fast_histogram outputs data.
To show the color bar just add plt.colorbar() before plt.show() .