What can Data Scientists Learn from Clickstream and Exhaust Data?
By: Laura Lewis
Tags: Machine Learning, Analytics, Data Science, Data

I’ve been thinking a lot about how useful exhaust data is recently, and wanted to share some thoughts. Below I set out what exhaust data is, some examples of how it can add a huge amount of value, and some practical suggestions for data scientists on what to use it for.

Exhaust data

What is exhaust data and why should I care?

Every company with any kind of online footprint will have exhaust data cluttering up its data stores somewhere. Exhaust data is data that is the byproduct of users’ activities. It doesn’t include ‘primary’ or ‘core’ data, like a user’s saved log-in details or their transaction history, but does include all the data that is produced as a side effect of how the user interacts with your digital product.

For example, I used to work as a data scientist at Citymapper, a transport app. You can use the app to get transit directions, and you can use a subscription ticketing product called Pass to pay for your journeys. Let’s say you wanted to travel from your home to your favourite communist dictator-themed pizza and cocktail bar (pre-Covid, of course). You’d open up the app, type in the address, look through some of the transport options, choose one, follow the directions and use your Pass to pay for the journey. Data about the transaction would be recorded, and would count as core data because paid transactions are core to the business function.

An example of getting Citymapper transit directions, which produces exhaust data. Source: The Verge

Event logs from all the other things that you did in the app would also be recorded — including what start and end address you typed in, which options you were shown, which options you clicked on, what the last option you selected was; and if you used the step-by-step route guide (‘Go mode’) your GPS location would also be recorded every 10 seconds. All of this is exhaust data. Similarly, exhaust data at an ecommerce site would include clickstream data — including every product you clicked on, how long you stayed on each page, which sections of each page you looked at, and what actions you took.

Exhaust data is big. Really big. It often has to have rules applied to it to stop it getting even bigger. In the Citymapper example, we could have recorded user locations every second in Go mode instead of every 10 seconds, but then we’d have 10 times as much data, without necessarily getting 10 times as much value. It can also be messy and difficult to work with; especially in terms of distilling it down into something more understandable, but also in terms of simply being able to access the data you need in a timely manner without your notebook hanging or your laptop exploding.

A love letter to trash data

Exhaust data is generally a massive, complex, nuanced and highly unsexy pile of data, which poses major challenges when trying to derive any kind of insight from it. It’s also wonderful, fascinating, and the most amazing source of information about user behaviour. It’s often underutilised because of the inherent difficulties in using it, so it can be a great untapped source of value for an organisation.

Exhaust data is enormously useful because it tells us not just what a person did, but how (and maybe even why) they did it.

I first realised just how important big, messy data that is produced as a byproduct of a core activity can be in my PhD, in the highly obscure area of the application of statistics and data science to the archaeology of human evolution. It’s only looking back on my doctoral thesis now, with the benefit of having spent several years using data science in the real world, that I realise that my whole PhD was about the importance of exhaust data.

Previous work in my field had tended to focus on data about the end product — in my case stone tools, but in my examples above this could be the final transaction on a transport subscription card or an ecommerce site. But by looking at the exhaust data I was able to learn so much more about human behaviour. In my PhD the exhaust data was literally trash data, as it came from measurements of bits of stone that were removed as a byproduct of making, for example, an arrowhead. But in a commercial context this could be the event logs that record each tap a user makes on an app before choosing their journey route, or the series of products a user looked at on an ecommerce site before making their final purchase.

Exhaust data is enormously useful because it tells us not just what a person did, but how (and maybe even why) they did it. And so, I’m writing this blog post as an homage to exhaust data.

Stone tools and statistics

What can stone tools tell us about human behaviour?

I finished my PhD in the Archaeology department of the University of Oxford back in 2015. I was interested in the evolution of human behaviour, and in particular when we started behaving in a recognisably ‘human’ way and how our species spread around the world. Bones and stones are generally the only things that survive from the time period I was interested in, and so I tried to answer these questions by looking at stone tool technology.

Traditionally, an archaeologist of the Palaeolithic or Stone Age would have a look at some arrowheads from two different areas, decide they look kinda similar, and then go and write a paper about how the same group of people must have lived in the two areas. This is, in fact, exactly how the previously predominant theory of our species’ dispersal around the world was born. This theory was based around the occurrence of a certain type of stone tool technology called microliths. As the name suggests, microliths are very small stone tools — things like arrowheads, stone barbs along the sides of spears, fishing hooks and teeny tiny blades.

Reconstructed microliths from Sweden around 9,000 years ago. Source: Larsson et al. (2017) in Lund Archaeological Review 22.

We find similar-looking stone tools in Southern Africa about 70,000 years ago, East Africa about 50,000 years ago, and South Asia about 40,000 years ago. Therefore, so the theory goes, early modern humans must have left Africa about 50,000 years ago and spread around the world taking their arrowheads with them. The story of early modern human development became reduced down to a map with a big arrow drawn across it saying ‘humans go this way’; a narrative whereby humans followed this big arrow from Africa to Asia and beyond, dropping arrowheads like breadcrumbs along the way.

Microliths I photographed at sites in South Africa (A and B) and Sri Lanka (C). Source: Lewis et al. (2014) in Quaternary International 350.

Enter archaeological exhaust data

I had a different idea — then heretical, now widely accepted and the subject of many papers and conferences. My idea was that, rather than looking at some arrowheads, deciding that they look similar and calling it a day, we should look beyond the finished tools themselves. Stone tools are the end result of a long process of bashing rocks together in just the right way, at just the right angle and with just the right force, to create the tool you want. In the process of making your lovely little arrowhead, you end up with a whole bunch of smashed up rocks that you don’t care about — or debitage, to use the fancy archaeological term (all the best archaeological terms are French).

Then, thousands of years later, an archaeologist in ugly beige trousers and a silly hat turns up and finds your debitage. She’s very happy to find your rock debris, because now she can figure out not just what you made, but how you made it. If the final stone tool is your core data, these leftover bits of rock are your exhaust data.

Me in my previous life as an archaeologist in a silly hat, being very happy to find rock debris, at Atapuerca, Spain.

Debitage can tell you a whole bunch about how exactly a tool was made. You can learn about what kind of stone a person selected, how they held it, how they hit it, what they hit it with, what their technique was, did they switch hammers, did they put it down and come back to it later, did they make other tools from the same rock, what did they care most about when making this tool, and a host of other aspects of their behaviour. Sometimes you can even put a whole core back together like a jigsaw puzzle and reconstruct the whole process (sometimes literally — a friend of mine used to leave heaps of debitage on a table in his department so that bored undergraduates would occasionally stop to put the pieces together), and sometimes there are gaps, but always there are useful behavioural insights to be gleamed.

Sure, finding an arrowhead might tell you that people were making arrows. But knowing exactly how it was made can tell you so much more about the behaviour of the person making it, and about their culture and traditions. Importantly, in the context of my research, it could tell me whether the same people were making the same tools in the same way, or different people were making the same tools in different ways. In biology this is called convergent evolution, where two species separately evolve the same trait because they’re dealing with the same environmental challenges, even though they’ve never even met to swap notes. For example, sharks and dolphins are both bad-ass intelligent sea creatures that hunt fish and have similar looking fins— but one is a fish and one is a mammal, and they’ve been evolving separately for about 300 million years.

The important take-away from this for data science is that it was only possible to figure out that different people were doing different things, despite the fact that the output was the same, by looking at data concerning the process they used to get there rather than just the end result.

Enter statistics

To skip to the end, after spending months in various parts of the world measuring stone tools and their associated stone debitage, and applying various statistical techniques and models to look for patterns in the data, convergent evolution is exactly what I found. People in South Africa and Lesotho were making very similar looking tools to people in India and Sri Lanka. But they were doing so in very different ways, and probably for different reasons — for example, hunting monkeys in the rainforests of Sri Lanka, and hunting antelopes on the South African savannah.

The important take-away from this for data science is that it was only possible to figure out that different people were doing different things, despite the fact that the output was the same, by looking at data concerning the process they used to get there rather than just the end result. Now that I’m a data scientist working in industry rather than an obscure corner of academia, I’ve realised that this is a very important lesson for thinking about an organisation’s data more generally.

What does all this have to do with exhaust data in the real world?

Applying the same approach to commercial data

The exact same approach can be used by companies and other organisations to extract value and determine insights from the exhaust data they store about their customers. Maybe you and I both went on the same ecommerce site and bought a copy of my book about my doctoral research (although I highly doubt it — it’s extremely niche). Looking at the core data alone, we’d show up the same, as our final transaction on the site was to buy the same book.

But maybe you went straight to the site, typed in the name of the book, went to the product page, bought it and left the site. Meanwhile, maybe I went to the books section of the site, looked at lots of different product pages, spent longer on each page, read all the reviews, got briefly distracted by a glow-in-the-dark marble run, then came back to the book and finally bought it. Maybe both these processes are pretty representative of our shopping and browsing habits. We are very different types of customer with different shopping habits, despite having ultimately bought the same item. Just as above I talked about different people making the same tools in different ways, we are different people buying the same product in different ways.

Practical suggestions for getting value out of exhaust data

There are all kinds of useful insights you might get from looking at exhaust data like this. For example:

These habits might be representative of broader categories of shoppers, and so some kind of cluster analysis for customer segmentation (a type of unsupervised machine learning) might help you to understand your customers better — perhaps through some effective data viz . It may also help your marketing department with creating user personas.

for customer segmentation (a type of unsupervised machine learning) might help you to understand your customers better — perhaps through some effective . It may also help your marketing department with creating user personas. Or they might allow you to help product management figure out what different groups of users need, and what features should be prioritised. For example, in my example above, I might be spending a lot of time on different product pages because I’m struggling to discover exactly which items I’m interested in — so perhaps some form of product recommender system might be in order, or improvements to an existing system.

might be in order, or improvements to an existing system. Conversely, in the example above, you might only use the site to buy specific things you have already been recommended by other people (maybe I told you all about my book in the pub and you promised to buy it to stop me talking about archaeology). That’s great, but maybe there are other items on the site that you might like but you don’t know about them yet. A data scientist might want to try some AB testing to experiment with different pages on the site, to see whether making some changes to the layout or content can encourage customers to spend longer on the site and check out some other products they might like.

Summary

Taking the time to clean, understand, extract and analyse exhaust data can be enormously valuable for a company. It can help you understand your customers on a much more granular level — not just what they do with your product, but how exactly they do it. Data science is at its best when it is directly helping customers. Even if working with exhaust data can be a little exhaust-ing (sorry), taking all of a customer’s data into account is a great way to really understand their behaviour, and deliver them the best possible products.