D3 + DS: Part 3
By: Leon T Johnson
Tags: Data Visualization, JavaScript, Hugo, D3js, Static Site Generator

D3 + DS: Part 3

Go Hugo!

Introduction

As you may have gathered from the title, I’m going to do things a bit differently here than what is provided in the book. That is, in the book, Murray gives instructions on how to host a website on a local server whose sole purpose is to show your D3 visualizations on an ad hoc webpage. I thought I could be a bit more practical about it, as I’d like to showcase what I do to the world. So, to do so, I applied a bit of what Murray did and a bit of what I’ve done in the past to show these D3 outputs using Markdown (and HTML) in my GitHub hosted Hugo blogs.

Setting up D3 for Hugo

Now, I’m not sure if this will work with Jekyll, but it is at least akin to what you would do on a traditional static site using traditional HTML coding:

Go to d3js.org and scroll down to the bit where you can see an HTML script tag like this <script src="https://d3js.org/d3.v5.min.js"></script> . Copy and paste whatever text is there (your version may be different from mine here, i.e., the .v5 ) into the head.html (or footer.html) partial for your Hugo webpage.

. Copy and paste whatever text is there (your version may be different from mine here, i.e., the ) into the head.html (or footer.html) partial for your Hugo webpage. If there isn’t one already, create a js directory in your static directory in which to put all your D3 JavaScript files. I created a sub-directory within my js directory called d3pds (for this D3 p lus DS blog), but you can do whatever you want.

lus blog), but you can do whatever you want. Within your JavaScript directory or sub-directory, add a .js file containing something like the below code in it. I called my first file d3pds_0.js, for reference. (I like to keep my JavaScript separate from my other code, just for the sake of cleanliness, organization, and versatility).

d3.select("#mything")

.append("p")

.text("This paragraph was added to the DOM using D3!");

Finally, in the Markdown file of whichever post you plan to put in some D3 visualizations, put the following tags

<div id="mything"></div>

<script src="/js/d3pds/d3pds_0.js"></script>

Of course, the mything here is in reference to the same #mything from the D3 code above (we’ll get into ID tag referencing later) and the script is telling the browser where to find that same script. Anyway, you can run hugo server or ./deploy (from the provided script in the setup page) and you should be good to go!

Alternative Simple Server (on MacOS or Linux)

If you’re not into doing this Hugo thing, and you just want a simple fix to play around with D3, try this:

Make a directory (say, myproject ) which contains (1) an index.html file like this one and (2) the d3.js file (or, the “minified” version d3.min.js, which takes out spaces). Open up your terminal and type python -m http.server 8888 & . for Python 3.x, or python -m SimplHTTPServer 8888 & . for Python 2.x. Navigate to localhost:8888 in your browser, and there ya go!

Updating HTML With D3

When I run hugo server , the above code (and process) renders

screenshot from my blog

Murray gives a pretty in depth discussion of what exactly happened here, but I’ll try to explain in my own words:

If you look at our code, we have a string of methods (functions) connected by . s. This technique is called chaining, and it’ll be the way we write most of our D3 code (of course, we don’t use this syntax all the time, just most). We start out with d3 which refers to the JavaScript library containing the methods (functions) to follow. Then, we select the element with ID mything . When you use select , as with most methods in D3, we are returned a reference to that element in the parenthesis. Once we “have” the mything element, we append a new p tag, which again returns a reference to that element. Now that we have that element, we update the text inside (between the <p> and the </p> ) to read “This paragraph was added to the DOM using D3!”

In the end, the D3 API (yes, daunting I know!) will be our friend when it comes to sussing out what returns what, and what needs what parameters.

DISCLAIMER: Scott Murray &c put a lot of work into creating an accessible and thorough instruction for using D3.js — I do not pretend to do the same here. On the contrary, I would like to use his instruction and apply it to something different (i.e., data science projects). Since the D3 material from the book is getting more and more dense, and since I do not have a whole lot of time to write on both data science fundamentals and D3, my explanations for what is going on with D3 will be far less detailed than what is in the book, but will be as comprehensive as possible. That said, you may need to do further research on your own to get a more complete and high fidelity grasp on the whole of the D3.js framework.

Loading Data

Alright, the moment we’ve all been waiting for: data stuff!

CSV logo! Boom.

Loading CSVs

When we load CSVs, D3 reads them row by row, where each row is an object (observation) by itself. So, if you have a dataset of n rows with the columns Country_Name and Country_Code , you’ll find that D3 reads a CSV as an array of n objects, each with Country_Name and Country_Code attributes.

That all sounds fine, but when it comes to CSVs, there are just a few problems:

For CSVs, all the data is read in as string (even if it’s numeric), which means you need a transformation function to turn numeric-strings into just floats/integer values using some initFunction (below).

Scott Murray’s .csv notation is deprecated. In Murrays book, which is based on D3 v4, he uses the syntax:

var initFunction = function(d) {

d.var = parseFloat(d.var); // converting strings to floats

d.var2 = parseFloat(d.var2);

} d3.csv("filename.csv", initFunction, function(data) {

console.log(data);

});

This is deprecated. Now, the fetch functionality (i.e., loading CSVs and JSONs, etc.) of D3 uses promise notation:

d3.csv("filename.csv").then(function(data) {

data.forEach(function(d) {

d.var = parseFloat(d.var);

d.var2 = parseFloat(d.var2);

});

console.log(data);

});

(notice the .then ; .csv used to be asynchronous, and now it is not.)

But really though … Here’s the thing, in practice (data science practice, that is), most of our work is done in Python, unless you’re an R-er, in which case, the following point still stands. Sometimes you do have a “final” CSV file saved to your computer for a one time visualization, forever and always. In this case, the above code is helpful in binding data for your viz. But that’s not super useful for realtime, dynamic use. Really, you want to have the ability to transform data coming in from some active or changeable database (or API), and then share it between a Python script and a visualization — true interactivity. In this way, data will be transferred via JSON objects. We’ll go over Flask apps and sending/receiving JSON objects later, but for now let’s just save/load our data to/from JSON files.

JSON logo!

Loading JSONs

If you’re working with data in your Jupyter(Lab) notebook(lab?), you’ll end up at a place where you will probably be saving the data using Pandas. If you’re going to be saving this as a JSON for use with D3, use the syntax

df.to_json("filename.json", orient='records')

The orient portion is important, so that D3 gets what it is expecting (i.e., an array of objects, where each is another observation). Then, you load the data using something similar as before (again, remember D3 fetching is no longer asynchronous, so we use promise notation).

d3.json("filename.json").then(function(json) {

console.log(json); //Log output to console

});

Binding Data

So, there are a few discrepancies in Murray’s book and the current v5 version of D3 (which is to be expected, since Murray is using D3 v4). The biggest one I’ve seen so far is closely related to the above case of .json and .csv being no longer asynchronous, but instead returning a JS promise.

Murray’s (v4) syntax resembles something like this:

var dataset; d3.get(file, initFunction, function(data) {

// do stuff ...

dataset = data

}); d3.select('thing')

.data(dataset)

. ... do stuff

From what I can tell, the best way to go about this now with v5 is more like this:

// Load the data

d3.json("filename.json").then(function(json) {

update(json);

}); // Bind the data and let it dictate an update

function update(dataset) {

d3.select("thing").selectAll("p")

.data(dataset)

. ... do stuff

});

The way I’m thinking about this is based on the fact that a JS promise is eager, but it doesn’t get fulfilled until it is instantiated by the browser. Basically, when I look at the // Load the data part, I just see it as setting the stage for the actual loading, basically like a pipeline. That is, we can interpret that first portion d3.json... as saying “I’m looking at filename.json, and I know exactly what to do with it (the function ) once you’re ready.” Whereas the second part is just the thing that it’s supposed to do when ready, i.e., bind the data dataset , in some way defined by …do stuff , to the p s in thing . Then, make the update once the browser calls it to.

If you like, check out my post on .enter , .exit , and .merge for a slightly more detailed discussion on what is going on with D3 data joining.

What’s Next?

That’s all for now, next up, we start visualizing with D3! Yay!