Plotting population data with d3 (1 of ?)
By: Khalifa S
Tags: Web Development, D3js, Graph, Data Visualization, Nodejs

Have you ever wondered how the population of a country has grown over the last few decades? Ever wondered how you would represent that on a graph? All while consuming an API? Well, have I got some content for you.

In this tutorial, I will walk you through the use of D3 — the most popular data visualisation and document manipulation library. We’ll also look at how to use JSON data to build a graph plot as well as set up a proxy to avoid the CORS (cross origin resource sharing) error. Finally, we will add some interactivity and styling to our page so it doesn’t look like it was made in the 90s. By the end of this tutorial, we will have the following result:

Ready to dive in? Let’s begin with the starter project.

Note: I’m assuming basic knowledge in node and JS.

In this project, we’ve got node and express set up. For the front end, there’s an index.html file within the public directory which we will use to display our plot. Let’s focus on the backend now.

Backend

We could definitely do without node for this project. However, I thought it’d be a great chance to explore the mechanics of having a service interacting with your frontend. Also, there’s an interesting challenge I would like to explore. Let’s focus on the first (and only) route in index.js. We want this to return some JSON data that we will eventually use for our graph plot. The data I’ll be using will be from The World Bank. They have an endpoint for getting population data for various countries. I’m using Nigeria’s population data, but feel free to use any other country (change the country code from NG to whatever country code you would like to use). Now you might be thinking, why don’t we just use the Fetch API within client side JS to get the JSON back? Well, let’s try it. Open a script tag in index.html and perform the fetch:

Save the html file and open it up in a browser with the dev tools open. What have we here?

This is the CORS error I told you about earlier. For an in-depth explanation of what this is, I’ll refer you to this post. However for our purposes, let’s just move on to the solution. What we need to do is set up a proxy that will make the request on our behalf and add an Access-Control-Allow-Origin header which will solve our CORS error. In our data route, we do the following:

Now if we run this server and access the data route, we have our data. Voila! Some of you might’ve noticed my use of a ternary for a query param above. We’ll address that in the front end. We can leave this server running in the background while we focus on the frontend. Onwards!

Frontend

For the sake of simplicity, we will add all JS and CSS in the index.html file. In future posts, we will separate our concerns as this project starts to grow. If you wish, you can do that now.

We need to create a function that will get data from our endpoint. We need this function to take an optional country code argument. We will use this in the future when we want to request for a different country’s population data.

Make sure to add the above code below all the script imports otherwise you might run into dependency issues later on. If you save and refresh, you will see the JSON data in the console.

You will notice that an array is returned with two elements. We’re only interested in the 2nd element which has all the population data. Note that the value and date values are strings. Now that we have a way of getting the data, we can do the fun stuff.

D3 — A gentle guide

Here’s the thing: d3 is an amazing library that allows you to accomplish more than just data visualisation. You can manipulate the DOM, handle events, create SVG elements and much more. The reason why it’s so great is also the reason why it’s a challenge to master. I will lay out a number of steps for you to follow which you should definitely practice replicating after this tutorial.

Add a div for where the graph elements will reside. I’m giving my div an ID of graphArea. Create a function that will handle displaying the graph with the supplied data. I’m calling mine drawGraph. Within that function, define some constants for your graph dimensions. I chose these values with the final result in mind. You could experiment with these to see what results you get.

const margin = { top: 30, right: 30, bottom: 70, left: 90 }; const width = 820 - margin.left - margin.right; const height = 500 - margin.top - margin.bottom;

4. Define an SVG area for where your graph will be drawn. The code below selects the div of ID graph area, adds an SVG element, provides a width and height with the constants we defined, appends a group and performs a translation to move that group such that the defined margins are respected.

let svg = d3.select("#graphArea") .append("svg") .attr("width", width + margin.left + margin.right) .attr("height", height + margin.top + margin.bottom) .append("g") .attr("transform", `translate(${margin.left}, ${margin.top})`);

5. Define the x and y axes. Let’s start simple. We need to define the x axis with a domain and range. You can think of the domain as the scale of your data. Say you had a list of ages from 1— 100, 1 would be your minimum and 100 would be your maximum (no kidding). Your range would be how you wish to represent that data on the axis. We have a pre defined width, so we would like the scale to respect the width of our graph. So our range would be from 0 to whatever width we set. When we define the axis, we must make a call to d3 to create this axis for us. This same principle applies to our y axis, except we would be looking at the height, not width.

let x = d3.scaleLinear() .domain([1, 100]) .range([0, width]); svg.append("g") .attr("transform", `translate(0, ${height})`) .call(d3.axisBottom(x)); let y = d3.scaleLinear() .domain([0, 50]) .range([height, 0]);

svg.append("g") .call(d3.axisLeft(y));

There are different kinds of scales we can use for our axes. To keep this tutorial short, I’m going to use scaleLinear and hope you will explore these scales in the docs. After adding this code to our draw graph function, save the file (remember to call the function within the script) and refresh the browser. You should see the following:

What a beauty huh?

Unsurprisingly, we don’t have any data to plot or axis titles because…we didn’t specify those. Let’s move on.

6. Redefine the scales using the data. This step depends on the kind of the data you’re looking at. In our case, we’ve got dates we want on our x-axis and values on our y-axis. Why? Ask your neighbourhood statistician. We’re gonna need to tweak the axes we defined to use the data.

let x = d3.scaleLinear() .domain(d3.extent(data, d => parseInt(d.date))) .range([0, width]);

The change made is the domain. d3.extent is a function used to return the minimum and maximum of an array. You specify your array, followed by an optional function that allows you to specify where these values are — in our case, each object has a date field we’re interested in.

let y = d3.scaleLinear() .domain([0, d3.max(data, d => parseInt(d.value)) * 1.2]) .range([height, 0]);

Similarly, we’ve made a change to the y domain. Notice I’ve used an array with 0 and d3.max. This function gets the maximum value in an array (duh). I could have used d3.extent, however I want my graph to start from 0. Also, I wanted to show that there are multiple ways to handle the domain. The * 1.2 you see is so the y axis is slightly larger than it should be. Take it out to see its effect. Now if we refresh the browser…the graph disappears? Look into the console. We’re getting a TypeError because we’re trying to access data when there is none. We need to make a call to getCountryData and then call drawGraph.

getCountryData("NG", (data) => { drawGraph(data); });

Now if you refresh, we should have our graph…with no ticks? What’s going on? Let’s whip out the debugger! Right-click on the page and click Inspect (assuming you’re using Chrome/Firefox) and navigate to the Sources tab. In the getCountryData function, put a breakpoint on the callback line like so:

Refresh the page and you should break on that line. On the right hand side, we’re concerned with the scope section. You can see the data array returned from our proxy service.

Taking a look, we have 2 elements. We only care about the element at the first index. Our graph isn’t showing because we assume that our data is actually that first element. We can either fix that client side or server side. I choose the former. Simply tweak the callback in getCountryData as follows:

callback(data[1]);

Take out the breakpoint and refresh the page. Voila!

7. Plot the data. Now the moment we’ve been waiting for. Sorry for the delay. Add the following to the end of your drawGraph function:

Refresh the page and Bob’s your uncle, you’ve got a graph! But we’ve got a problem…before we get to that, let me explain the code above. I grab the svg element, append a group, followed by a path, specify the (static) data, set fill to none, stroke to lime, stroke width of the path to 1.5 and finally the x and y coordinates for the path. I encourage you to tweak these attributes to see the effect.

Now back to the plot. It seems past 2015, the plot drops sharply to 0. What gives? Back to the debugger!

Take a look at the first 2 values returned in the data array. You can see 2019 has a value of null (probably because there’s no population data for that year) but 2018 has a value. What we could do is simply run shift() on the array before passing it to the callback. But what if there’s more null values later in the future? Instead, we’ll just remove all null values using the array filter method. Observe:

callback(data[1].filter(d => d.value));

Remove the breakpoint and refresh your page. Much better!

8. Provide titles for axes and graph. Our graph wouldn’t be much good if people didn’t know what they were looking at would it? Let’s provide titles for our axes. In drawGraph, add the following:

I’ll save you the headache of figuring out where to position those titles. Tl;dr the x and y attributes are what you use to position the title. In the case of the “Year” title, we want it to be horizontally aligned so we use transform to move it to the halfway point of the graph and push it down slightly so it doesn’t interfere with the y ticks. For the “Population” title, we rotate it because it’s a vertical axes. When we rotate it, the x and y axes flip accordingly so we need to account for that in the calculation of the x and y positioning.

Interactivity

At this point, we’ve got our graph and we could simply create a styled page around this and call it a day. However, it’s not very user friendly. For example, how do we know what the value is for the year 1973? Of course we could eyeball it but we want the precise number. This is where the use of a tooltip comes in. Most visualisation tools you see have the ability to show you more information once you hover over a given area. We shall have that same functionality. Let’s start by defining our tooltip.

let tooltip = d3.select("#graphArea") .append("div") .style("opacity", 0) .attr("class", "tooltip") .style("background-color", "white") .style("border", "solid") .style("border-color", "#64FF8B") .style("border-width", "1px") .style("border-radius", "5px") .style("padding", "10px");

We define a tooltip in the graphArea and set some styling. We could remove styling and put that in our css, however I want everything to be in one spot for now. Now we need to implement event handling on our graph so we can update the tooltip accordingly. What we would like to do is make the tooltip visible when we hover over the graph (mouseover), update the values when the mouse moves along the graph (mousemove) and hide the tooltip when the mouse exits the graph (mouseleave). Let’s start with mouseover (in drawGraph).

let mouseover = function(d) { tooltip.style("opacity", 1) };

Now where do we attach this event handler? We need to create a rectangle on our SVG area such that it takes the job of handling the mouseover event.

svg .append('rect') .style("fill", "none") .style("pointer-events", "all") .attr('width', width) .attr('height', height) .on("mouseover", mouseover);

Now if you save, refresh and hover over the graph, you should see the tooltip at the top-left hand corner of your browser window.

It’s currently stuck because we’re doing nothing to update its position. Let’s move on to mousemove. For now, update the above code with:

on("mousemove", mousemove);

Then define the mousemove function (before the call of course):

let mousemove = function(d) { console.log("Hiiii"); };

With the console open, save and refresh the page. Then hover over the graph. The console should be spammed with the console log we created. This shows our mouse is being tracked. But how do we get access to its position? Update mousemove as follows:

let mousemove = function(d) { const x0 = x.invert(d3.mouse(this)[0]); console.log(x0); };

The above code will get the x position of the mouse from d3.mouse and invert it using the x axis we defined for our graph. Refresh and view the console. You should see several lines of what looks like year values. This is because the x axis is…year values. Go figure.

Now what we want is to get the years as integer values and not floats. We’ll do a little bit of math to clean up the data.

let mousemove = function(d) { const x0 = x.invert(d3.mouse(this)[0]); const decimal = x0 - parseInt(x0); const i = decimal > 0.5 ? parseInt(x0) : parseInt(x0) - 1; console.log(i); };

The above code with check whether to stay with the current year or fall back to the previous year based on the decimal point value. Save and refresh to appreciate the mathematical brilliance.

Now that we have the year, we can grab the corresponding population value and display it. For this, we will use the array find function.

let selected = data.find(e => e.date == i); if (selected) { tooltip .html(`Date: ${selected.date}, Population: ${(selected.value)}`) .style("left", (d3.mouse(this)[0]+ 90) + "px") .style("top", (d3.mouse(this)[1]) + "px"); }

The above code will find our value and append it to our tooltip using template strings. The positioning is used to set where the tooltip will be on the screen. Save and refresh and boom! You’ve got a working tooltip!

While our tooltip is in fact showing, we have no way of getting it off the screen. We will resolve that by adding mouseleave.

let mouseleave = function(d) { tooltip .transition() .duration(200) .style("opacity", 0) }

This will “remove” our tooltip from display. Now where do we attach this event handler? Let’s try the rectangle the element we used for mouseover and mousemove. Refresh and save…huh. That didn’t work. How about the svg element?

let svg = d3.select("#graphArea") .append("svg") .attr("width", width + margin.left + margin.right) .attr("height", height + margin.top + margin.bottom) .append("g") .attr("transform", `translate(${margin.left}, ${margin.top})`) .on("mouseleave", mouseleave);

Save and refresh…huh, that works? Why is that? I obviously knew that would be the case but I just wanted to show the difference. We put mouseleave on our svg element because that’s the whole graph area. Our graph is contained within there, so if we exit the graph, then and only then should we see it leave.

So at this point, we’re done with the graph! Now we can move on to adding some styling to the page before we wrap up this tutorial.

Styling

This section is actually gonna be pretty brief. As we go through the series, we will incrementally develop the UI. However, you may wish to do some work yourself. I’ve included Bootstrap and its dependencies to index.html so we’re just gonna add some elements and classes to the page.

<nav class="navbar navbar-expand-md navbar-dark bg-dark"> <a href="#" class="navbar-brand">Dank Data</a> </nav> <main class="container"> <div id="graphArea"></div> </main>

Save your index.html and refresh. You should have a fancy navigation bar and your graph in a container. If you full screen your browser, you should notice that the graph isn’t perfectly centred. Let’s fix that. Add a style tag to your html with the following styles:

#graphArea { display: flex; justify-content: center; }

Save and refresh and that should do the trick.

Conclusion

As promised, we have created a graph that can be used to visualise population data and just about anything!

Now I’m aware that this won’t make it to Bloomberg, but the whole point of this was to show the absolute basics to data visualisation and cover the parts that tripped me up when I was getting started so you won’t have to make those same mistakes.

In the next tutorial, we will look at plotting multiple paths on a graph to show the comparison of various countries as well as giving users more interactivity with our snazzy graph. Till then, stay safe!

To access the finished product: https://github.com/khalifa-is/d3-pop-data-tutorial-finished.