Dashboards in Your Browser: Lessons From Working With d3.js
Tags: Flask, D3js, SVG, D3, Data Visualization

When learning a new data tool, I like to work with football data, since I know and understand it reasonably well. This example uses data from the 2017–2018 Premier League season provided by Football Reference.

Lesson 1: Pass only the data that you require to the web page.

This example uses Flask as a back-end. Flask is a micro web framework written in python. I won’t go into detail about Flask in this post; in this case it renders, and passes data to, the web page.

Why am I using Flask?

Your data takes up memory. The more memory your webpage requires, the longer it will take to render and to function. And yes, I learnt this by trying to dump my entire data file into my d3 viz and waiting forever to see the goods.

Using Flask, I’m able to utilize forms in my web page to request only the parts of the dataset that I require. In our example, we’re only ever comparing two Premier League football teams at a time. Hence, I only need to pass data for those two teams to the web page:

Example d3 visuals https://github.com/JuniorSen/D3-filters-update-cycle

I value versatility. If you’ve spent more than a few hours angrily trying to learn how to use d3 (it’s the only way), then you will too.

Lesson 2: Buttons are your friend

Using buttons, one chart can become many. You’ll have noticed in the gif that by clicking on the ‘Goals’ or ‘Assists’ buttons, the variables presented on the scatter plot are changed and the axes updated. Here’s how they work:

d3.select(“#goalscoringBtn”) // Find the button in our HTML code

.on(“click”, function() { // When button is clicked, run function

xVal = ‘Goals’ // Change axis value variables

yVal = ‘xG’

updateChart(xVal,yVal) //Call update function with new values

}); d3.select(“#creatingBtn”)

.on(“click”, function() {

xVal = ‘Ast’

yVal = ‘xA’

updateChart(xVal,yVal)

});

Simple!

Lesson 3 (a lesson in life): Don’t try to reinvent the wheel

I had an idea. I wanted to be able to see data from different periods in the season. Standard date range filters bore me. More importantly, I can’t see what was going on over the period I select if all I have is a date range.

I’d seen some examples of people using d3.brush to zoom in visuals. I liked the idea of being able to see the results over the course of the season and just being able to click and drag over those I wished to include. You can see where this is going: Brush filters! Here is how the brush filter works:

var extent = [0, 10000] // This sets the default range to include

// all dates in our dataset svgForm.append(“g”).attr(“class”, “brush”)

.call( d3.brushX() // Add the brush feature using the d3.brush

// function on the x axis only

.extent([[0,0], [formWidth,formHeight]]) // initialise brush area

.on(“end”, updateAll) // Each time the brush selection changes,

// trigger the ‘updateChart’ function

).lower(); // pushes the brush down a level (behind the dots) function updateAll() {

updateChart(xVal,yVal); // Call the update function with current

// axis vars

updateChart(xVal,yVal);

}

You have a question. Why am I calling updateChart twice?

Frankly, it’s a bit of a workaround. When I use the brush filter, updateChart is called and all proceeds as one might expect. Except if there are dots to be added. The second call appends these fresh dots from the enter selection.

Implementing the brush filter means that we need date-level granularity in our dataset. We want to present this data as an aggregate through our final visual.

Lesson 4: Grouping data — d3.nest

Grouping is actually quite straightforward using d3. You’ll use the following functions regularly. Save yourself a copy so that you can slot them straight into future projects.

Anyway, specify a key to group the data by and rollup any fields you’d like to include:

//nest main dataset by player

var nested = d3.nest()

.key(function(d) {return d.Player})

.rollup(function(player) {

return player.map(function(c) {

return {“Gls”: +c.Gls,’Min’:+c.Min, “xG”: +c.xG,’Ast’:

+c.Ast,’xA’:+c.xA, ‘DateParse’: c.DateParse, “Squad”:

c.Squad,”P_color”: c.P_color, “S_color”:c.S_color

}

});

}).entries(records) //input dataset

Next, you’ll need to sum the data in each group. Another handy function:

//Sum up stats in the nested data

function sumStatsScatter(node) {

if (node.value) {

node.values = node.value;

delete node.value;

}

node.Goals = node.values.reduce(function(r,v){

return r + (v.value? sumStats(v) : v.Gls);

},0);

node.xG = node.values.reduce(function(r,v){

return r + (v.value? sumStats(v) : v.xG);

},0);

node.xA = node.values.reduce(function(r,v){

return r + (v.value? sumStats(v) : v.xA);

},0);

node.Ast = node.values.reduce(function(r,v){

return r + (v.value? sumStats(v) : v.Ast);

},0);

node.Min = node.values.reduce(function(r,v){

return r + (v.value? sumStats(v) : v.Min);

},0);

return node.Goals, node.xG,node.Ast, node.xA,node.Min

}

//Call function on nested data

nested.forEach(function(node) {

sumStatsScatter(node)

})

There’s something missing here — we’re pressing buttons and applying filters and re-grouping data. Where are we doing that?

Lesson 5: Try to understand, or at least learn to use, the d3 update cycle

The process for updating a d3 viz when applying filters can be confusing at first. It’s worth taking the time to work through some examples and gain some understanding of what’s going on before you sink countless hours of frustration into winging it (à la this guy).

Update functions allow us to filter, transition and even redraw components of our visual. That’s powerful. Let’s go through the update cycle of our example step-by-step:

//Update function

function updateChart(xVal,yVal) {

extent = d3.event.selection //used to find selected date range

if(!extent){

extent = [0, 10000] //if no date range selected, take all

// available

} //Filter the dataset on the range selected by the brush filter:

var dataFilter = records.filter(function(d) {return d.DateParse >=

x.invert(extent[0]) & d.DateParse <= x.invert(extent[1])})

So, our update function has been called. First things first. Get your data in order. Using x.invert(extent…) we can convert our brushed range into a min and max date. Filter your data on these dates and re-group with a fresh call of d3.nest.

Next, get those axes in order: find your new data range, redefine your scales and redraw your labels. Here’s a snippet of the x axis update:

// Find the new max x/y value. I want both axes to have the same

// range, so I find the max of the two axes.

var scatterDomMax = d3.max(nestedFilter,function(d){return (d[xVal]

/ (d.Min/90))})*1.1 > d3.max(nestedFilter,function(d){return

(d[yVal]/(d.Min/90))})*1.1 ? d3.max(nestedFilter,function(d)

{return (d[xVal] / (d.Min/90))})*1.1 : d3.max(nestedFilter

,function(d){return (d[yVal]/(d.Min/90))})*1.1 // Redefine define x scale

var xScale = d3.scaleLinear()

.domain([0,scatterDomMax])

.range([margin.left, 400]); // Call your new axis with a transition

var xAxis = svgScatter.selectAll(“g.x-axis”)

.transition().duration(500)

.call(d3.axisBottom(xScale)) // Remove old axis label

svgScatter.selectAll(“.scatterLabels”).remove(); // Add new axis label

svgScatter.append(“text”)

.attr(“class”,”scatterLabels”)

.style(“text-anchor”, “end”)

.attr(“x”, 405)

.attr(“y”, 395)

.style(“font-size”,”24px”)

.style(“fill”,generalTextColor)

.text(xVal + ‘/90’).lower(); // pushes text back so that we can

// still hover over dots

Finally, let’s talk about the enter-exit-remove update cycle of d3:

// Select all of the circles currently on our scatter plot and

// update the data source

var updatedDots = d3.select(“#scatter-chart”).select(“#svg-”).selectAll(“circle”).data(nestedFilter); // Enter our new data and append any new circles required.

updatedDots.enter().append(“circle”); // Remove any circles which no longer have data bound

updatedDots.exit().remove(); // We will need to redraw our tooltip with updated data

var tooltip = d3.select(“#tooltip”).style(“opacity”,0); // Add a transition to move the dots to their new positions, add

// or remove dots as necessary

updatedDots

.transition()

.duration(1000)

.attr(“cx”, function (d) { return xScale(d[xVal]/(d.Min/90)); })

.attr(“cy”, function (d) { return yScale(d[yVal]/(d.Min/90)); }) // etc...

There’s quite a lot going on there. Let’s break it down step-by-step, as I understand it: