Structure Your Data Science Projects by Continuously Telling Stories
By: Paul Hiemstra
Tags: Productivity, Narrative, A series of articles on learning an AI bot to play tic-tac-toe, Data Science, Project Management

Reflection is king

One of the biggest risks I see in a data science project is that you continue doing research without reflecting on your progress. You do lots of work, but not necessarily solving the problem. What makes this kind of reflection hard is the fact that you need to lift yourself up from the nitty gritty of your work, and take a helicopter view. How do my results progress me towards my ultimate goal?

What really helps forcing a helicopter view is using a method such as CRISP-DM. This model cuts up the data science workflow into a number of phases, ranging from assessing your business problem and your data, to performing analyses and reflecting on what they mean for the business problem. The goal is not to just perform statistical analyses, but to craft a narrative that fully answers the business question. The organised flow of CRISP-DM reminds us to reflect often.

Embrace iterations

Via Agile methods such as Scrum, the IT world has embraced iterative working. In research this is just as, if not more important. Our project does not consist of one big reflection (CRISP-DM) cycle, but dozens of cycles that craft dozens of narratives. These smaller narratives form the backbone of the larger overall story.

Continuously refine and merge

Using reflection, narratives and iteration we can structure our data science project. In my experience, at the beginning of the project we start with a very basic understanding of the business problem. For example: how can we detect anomalies in x-ray images. From this basic premise we craft our first few starting narratives. One simple narrative could be that we ask the hospital for two sets of x-ray images, one that does and one that does not contain anomalies. We simply look with our eyes to see if we can tell the two sets of images apart. From that visual inspection we craft a narrative. In the evaluation part of the narrative, ideas for statistical tools to use can emerge as new prospective narratives.

As the project moves along, the narratives become more and more intricate and build on top of each other. In addition, some narratives are deemed dead ends and are discarded. If we write a research paper at the end, this presents the final overall narrative that wraps and summarises all the previous smaller narratives. The later intricate narratives often influence the final paper more strongly.

Planning narratives

But how do we plan the flow of all these interlocking narratives? We can try and borrow some concepts from Scrum. We can use a planning board to track our narratives, with the prospective narratives listed in the backlog. In the sprint review we can present and spar about the narratives, further honing their storylines. In addition, we can use the sprint planning to decide which of the narratives on the backlog presents the best way forward. Finally, in the Sprint Retrospective we discuss as a team how we can collaborate more effectively. Other staples such as fixed sprint length or using story points for narratives might not work very well, but we can still create a structured way of working in the hard to plan world of data science research.

Who am I?

My name is Paul Hiemstra, and I work as a teacher and data scientist in the Netherlands. I am a mix between a scientist and a software engineer, and have a broad interest in everything related to data science. You can follow me here on medium, or on LinkedIn.

If you enjoyed this article, you might also enjoy some of my other articles: