The sleep diary of a Data Scientist
Tags: Machine Learning, Data Science, Dreams, Data Visualization

The sleep diary of a Data Scientist

A data-driven analysis of my dreams

Hi!

My name is Hargun Oberoi, I am a mathematics graduate from India and having worked in the tech industry, like millions of people around the world, I’m grappling with what I can only describe as the biggest challenge of our century: A good night’s sleep.

Although I’ve been fairly active in sports, the last few years have been difficult for my health, as I moved into a new city post graduation, one choked with traffic, pollution, and all the stuff that makes any healthy lungs sick.

A ‘normal’ day in the city of Bangalore

This led to a host of breathing difficulties which peaked in the year 2018.

It made me realise how much we take the simplest things in life for granted! The epiphany aside, the biggest collateral fell squarely on my sleep and I would spend hours each night; at best, twisting and twirling, and at worst, gasping for air.

Doctors didn’t really seem to have much of an answer to this endemic, and the usual prescription would be an anti-histamine (or steroids).

This vicious cycle had a spiralling effect on my health; Lack of sleep meant regular medication, which affected my natural sleep cycle, leading to stronger doses. This went on for a while before I realised I had to do something about it.

That’s when I started the ‘Rise Before Sunrise’ project:

A questionnaire of four simple sleep related questions, to be answered almost immediately after waking up.

The Google Form

How could this help?

If I couldn’t change my sleep cycle, I could at least try to see what went wrong, and if my troubles were getting worse; A sort of a thermometer for my sleep.

Fast forward to the end of 2019, I had amassed nearly two years worth of dream data! With a little time and enough data on my hands, I decided to scope through it. Some of the more interesting finds are presented below.

Litmus Test

The basic data I captured had the following fields:

Did I get up before sunrise? (Yes/No) What time did I wake up? What rating would I give my sleep? A description of what I could remember from my dream

The raw data

I ‘knew’ subjectively about my struggles and challenges, and changing sleep patterns because of a few lifestyle changes, but would the data reflect that knowledge? I decided to begin with a few basic questions to verify.

Q1. How would my wake time compare between 2018 and 2019?

I ‘knew’ that in 2019, I was waking up less erratically than 2018 because of a few lifestyle changes. So let’s start with a plot of the wake-up time.

The graph above is a bar plot with the x axis showing the wake up time and the y axis the percentage of days I woke up at that hour.

The graph above shows a lower sigma value (statistically representing variation in data) for 2019 as compared to 2018.

It confirms that the variance of 2019 wakeup time is less than 2018.

So far so good! Next question.

Q.2: What about wake up time before and after August 2019?

In August 2019, I decided to move out of my previous job. Ever since that change, I could sleep in a little late, if I didn’t feel well rested.

Was this obvious in the data representing my subsequent rising times? Let’s see.

Wow! Now that’s a lot more revealing. A quick look at the stats shows that although my average wake up hour was much more, the variance was again much lesser.

That made sense because I was consciously trying to create a morning schedule, and even on my worst days, I could get out of bed without being too miserable. Second test passed as well!

Q3. But what about the weekends before and after August?

Weekday data may not be a true representation of one’s ‘actual’ wake up time. Ask any kid forced to wake up at 6 am to catch the school bus!

But weekends often lets many of us rise as per our wish or convenience.

After a few lifestyle changes post August, I felt confident that I was naturally rising more consistently, without alarms or my mom screaming in my ear.

So what does the plot say?

Yes! The wake-up time on average was fairly consistent, but there was a variance of an entire hour for weekends before and after August! The data makes sense.

If the stats don’t make it clear, have a look at the two largest blue bars which are within the two largest red bars, confirming what I felt about my sleep cycle changes. I was rising more consistently post August, even on weekends.

Dream Recall

Time to ask more interesting questions. What about the actual content of my dreams?

I had noticed that on some days, I remembered a lot of my dream, and on the others, almost nothing. Was there a connection between the rating of my sleep (which I scored on 5) and the amount of dream recall?

I drew up a simple regression to see a connection, with the amount of text in y axis, and the dream score in the x axis.

Indeed, dream score showed some correlation with amount of recall.

To visualise more clearly, I decided to make a violin plot. It’s a nice visualisation that lets you also imagine the spread of the points for a given y-value. i.e If it has a big belly for a given y-value, many points are identical.

Et voilà! Can you guess the relationship of scores with amount of recall?

It appears that although high scores are not significantly associated with high recall, low scores seem to be strongly associated with a poor recall.

I decided to look for a study comparing dream recall and quality of sleep and the very first result on google was this research study conducted in 2011.