How I Created an Interactive, Scrolling Visualisation with D3.js, and how you can too | by Cuthbert Chow | Towards Data Science

cuthchow.github.io
Visualisation
Code (Github Repo)
The web is littered with stunning examples of scrolling and interactive visualisations, often built using D3. If you’ve ever read such feature pieces from Bloomberg or the New York Times, I’m sure you know what I’m talking about. Some of my favourite examples of this form are:
Shirley Wu’s exemplary visualisation of very line in Hamilton, built with D3 and React. In fact, all of her work is excellent.
The New York Time’s piece of how the economy has been reshaped by the recession, told in 255 different charts.
This absolutely mind-boggling piece, visualising the inner mechanics of Machine Learning, created by the folks over at R2D3.
This quirky visualisation, breaking down the amount of water which we consume on a daily basis
Another article from the NYTimes, illustrating the treacherous climb up El Capitan’s dawn wall
And so many more.
Seriously, every single one of these projects is worth checking out and learning from. They are amazing.
There’s something satisfying about the way in which the visual elements comport themselves after each scroll event, bending to your control. Often times, I have a huge smile on my face as I scroll up and down the page, watching all the shapes and colours fly around. It’s a sensation reminiscent of discovering the secrets held by the data on your own, which is what I believe makes these presentations so convincing. It also provides a much more commanding narrative flow, which would otherwise be lacking in more static pieces. Browsing through these articles, it may appear that such visualisations would be extremely complicated to produce on your own, and you may assume they are best left to the editors and writers of top-shelf publications. But I’m here to help dispel this notion: Anyone can make one, including you!
After reading through a bunch of online resources (I will leave a list of these at the end of the article), I decided to take a stab at creating one myself. As I’ve mentioned before, the best way of learning a new tool, technique, skill or ability is to put it to use, before you even know if you’re capable of doing that thing. And so that is exactly what I did over the course of a week.
(This article will assume you have some basic familiarity with D3.js and how it works)
After browsing through a series of datasets, I settled on one which I found hosted on Kaggle, which originally came from the analytics blog Fivethirtyeight (if you’re looking for cool datasets, check out google’s dataset search tool). It was a (slightly outdated) dataset about college majors around the United States. It had information on the number enrolled in each college major, as well as the median, 75th percentile and 25th percentile salaries from those students. Interestingly, it also provided information on the male/female representation for each major, which I thought would be a good point of focus for the piece.
With the dataset decided, I started exploring the data using Python and Pandas in Jupyter notebooks, looking for any interesting trends or points I would want to emphasise in a visualisation. I also made rough, crude approximations of the graphs I would want to show in the final product.
My general conception of the visualisation was that the college majors would each be represented by a bubble. The bubbles would then change form and colour based on the type of data I was trying to highlight (ie gender discrepancies, median salary, category grouping etc.). When the bubbles would be grouped in by specific attributes, I wanted to achieve this through a D3 force simulation, so that the bubbles would appear to be gravitating towards an unseen force.
After some exploration, I knew that I wanted to focus on the impact that gender distribution within a major has on the median salary of its graduates. I would strongly suggest that you take a deep dive into your data before starting any of the visualisation process, because you want to have some semblance of the interactions you have in mind before actually sitting down to code the final product.
If you want a really quick way of observing relationships between variables, the python visualisation package seaborn has a ‘pairplot()’ method to help you do just this, which creates a matrix of scatterplots between all of the variables in the dataframe passed to it.
#Plotting a scatter matriximport seaborn as snsfactors = df[['Total', 'Major_category', 'ShareWomen', 'Sample_size', 'Employed', 'Unemployed', 'Median']]sns.pairplot(factors)
Now we have a solid idea of the things we want to include, we are ready to start creating.
This is (obviously) the most important part of every scroller you build. But conveniently, it’s also the most reusable part, meaning that after you build it once, you should be good to go.
In fact, I’ve created a template version of the scroller functionality, which you can find right here. (credits again to Jim Vallandingham, as much of the code used was sourced from his original article. In fact, he wrote a similar article on creating a scroller, with an emphasis on the scrolling mechanics, so you should read that first. In this article, I will focus on the elements which I found confusing, and put a greater emphasis on the part of actually creating the visualisation).
The entire visualisation should be contained within a single HTML page. Within the page, there is a div which contains all the elements, and within that div, there are two separate divs for the text and the visualisation.
Within the ‘sections’ div are a bunch of section elements, each with the class ‘step’. Each one of these sections should contain a distinct section of text, and correspond to one phase or part of the visualisation. It is when the user moves between sections that the visualisation will be updated,
<div id="graphic">    <div id="sections">         <section class="step"></section>         <section class="step"></section>    </div>    <div id="vis">    </div></div>
You can size and position these elements as you wish, so long as you keep the #vis element fixed on the page (using display: fixed), so it doesn’t move as the user scrolls. You can also tweak the height of each .step element based on how much text you want to include per section.
Here is the code which handles the scroll events. I’ve added comments to clarify certain parts.
function scroller(){  let container = d3.select('body')  let dispatch = d3.dispatch('active', 'progress');  let sections = d3.selectAll('.step')  let sectionPositions  let currentIndex = -1  let containerStart = 0;// Binds the position function to the scroll event, and the resize function to the resize event. What these functions do are detailed below.   function scroll(){    d3.select(window)      .on('scroll.scroller', position)      .on('resize.scroller', resize)      resize();    let timer = d3.timer(function() {      position();      timer.stop();    });  }//The resize function determines where each of the .step elements are on the page, relative to the top of the first element. It saves all of the co-ordinates of these elements in an array called sectionPositions  function resize(){    sectionPositions = [];    let startPos;    sections.each(function(d, i) {      let top = this.getBoundingClientRect().top;      if (i === 0 ){        startPos = top;      }      sectionPositions.push(top - startPos)    });  }//The position function determines where the user is on the page (using window.pageYOffset), and uses that to determine which section of text should currently be in view. It then uses D3’s dispatching tools to signal the 'progress' event, which will be used in the main script, passing along the current section index so that the script knows which stage of the animation/visualisation should be showing.   function position() {    let pos = window.pageYOffset - 300 - containerStart;    let sectionIndex = d3.bisect(sectionPositions, pos);    sectionIndex = Math.min(sections.size()-1, sectionIndex);    if (currentIndex !== sectionIndex){      dispatch.call('active', this, sectionIndex);      currentIndex = sectionIndex;    }    let prevIndex = Math.max(sectionIndex - 1, 0);    let prevTop = sectionPositions[prevIndex]    let progress = (pos - prevTop) / (sectionPositions[sectionIndex]   - prevTop);    dispatch.call('progress', this, currentIndex, progress)  }//The code here adds an event listener to the dispatcher.  scroll.container = function(value) {    if (arguments.legth === 0){      return container    }    container = value    return scroll  }  scroll.on = function(action, callback){    dispatch.on(action, callback)  };  return scroll;}
All of the above code is saved in a separate script file. We then create a main script file, which is responsible for updating the visualisation every time the viewer changes sections.
let scroll = scroller().container(d3.select('#graphic'))scroll()let lastIndex, activeIndex = 0//This is where most of the magic happens. Every time the user scrolls, we receive a new index. First, we find all the irrelevant sections, and reduce their opacity. scroll.on('active', function(index){  d3.selectAll('.step')    .transition().duration(500)    .style('opacity', function (d, i) {return i === index ? 1 : 0.1;});//Next, we selection from a range of activationFunctions (which we create), based on the index of the current section.   activeIndex = index  let sign = (activeIndex - lastIndex) < 0 ? -1 : 1;  let scrolledSections = d3.range(lastIndex + sign, activeIndex + sign, sign);  scrolledSections.forEach(i => {    activationFunctions[i]();  })  lastIndex = activeIndex;})//I placed all the functions in an array. Each function corresponds to a different change in the visualisation. One may change the graph into a scatter plot, and another may initiate a force simulation.let activationFunctions = [  draw1,  draw2,  draw3,  draw4,  draw5,  draw6,  draw7,  draw8]
With the generic components of the scroller sorted out, what you do from here is going to vary greatly based on the type of visualisation you are hoping to construct. Nonetheless, I can hopefully offer some broad brush advice that may come in handy, and which I wish I had been aware of sooner in the process.
The essential idea is that all the elements that you need for the visualisation should be created when the page loads. I created a function called drawInitial(), and invoked it once the data was loaded in. This involves creating all the scales, axes, shapes, for simulations and other elements which you will need in the visualisation.
Once they are all created, you can simply set their opacity attribute to 0 to ensure that they don’t show up until required by the visualisation. Make sure each set of features (ie a set of axes for a particular chart, or a detail callout) has associated class or ID names, so that you can reference those specific features when required.
I found it useful to create a cleaning function, which would be invoked at each step. I would pass in the chart type which I was trying to create, and the function would remove (or hide, rather) all the elements which were extraneous to that chart, thus saving me from having to write a lot of duplicated code. This is what part of the function looks like:
function clean(chartType){    let svg = d3.select('#vis').select('svg')    if (chartType !== "isScatter") {      svg.select('.scatter-x').transition().attr('opacity', 0)      svg.select('.scatter-y').transition().attr('opacity', 0)    }    if (chartType !== "isMultiples"){      svg.selectAll('.lab-text').transition().attr('opacity', 0)      svg.selectAll('.cat-rect').transition().attr('opacity', 0)...
As I mentioned, I would be using force simulations to group the circle elements in several of the phases. The key thing to remember is to ‘reheat’ the force simulation every time you bring it back into use. Force simulations in D3 have an alpha value, which is a cooling parameter that gradually decrements over time, and reduce how much the force impact the nodes. You should reset the alpha back to a certain value every time you restart the force, to make sure it has enough ‘energy’ to get to the desired state. You can also tweak the alpha decay rate to affect how quickly the simulation settles down to an equilibrium state.
This part is key: Always use transitions when changing the state of the visualisation, even if that change is something you want to be instantaneous. By utilising D3’s transition feature (even if the duration is 0), you allow the changes to be interrupted. This means that if the user is scrolling through the sections quickly, one transition can be overridden by a subsequent one, avoiding the case where you are left with an awkward, invalid state.
SVG elements are ordered based on when they were appended to the parent element. Think of it like throwing paint on a canvas: the last layer of paint is the one that’s visible on top. You could reorder elements by removing them from their parent element, and then appending them again to the very end. Alternatively, you could make use of D3’s .raise() and .lower() methods, which conveniently do this exact process for you when given a selection of elements.
Colours are an indispensable component of good data visualisation. For parts of my project, I was trying to encode with colour 16 different categories of college majors. This led to an elaborate process for selecting colours, as I wanted to avoid any two hues which were too similar and thus easily confused.
I highly recommend using Color Hunt in building your palettes. It’s a website where other creatives have constructed and shared their own palettes, and it allows you to easily copy the hex-codes of the hues which you want, and to save your own palettes as well for future use.
Hopefully, this has demystified at least part of the process of making a scrolling visualisation. It’s honestly much easier than it looks, and so definitely go find an interesting dataset and try it out for yourself!
Jim Vallandingham: article on scrollers, blog
Shirley Wu’s work, as well as her YouTube channel and Twitch streams
Mike Bostock, blocks, article on how to scroll