7 Months of a Relationship, WhatsApp Messages Analyzed | by Kiprono Elijah Koech | Towards Data Science

Communication in a relationship is key, so they say. Without communication, any relationship is as good as dead. In this era and age, technology as taken the center stage as far as communication is concerned. In this article, we will analyze WhatsApp messages between my fiance and I and at the end of the day you can apply the knowledge on your on WhatsApp data. We will use Python in this analysis and specifically deal with: Pandas Tables, Data Visualization, Data Cleaning and much more.
Before we go through this I will like to mention that we have been in this relation for a year and half now and the chats available are from Dec, 18 2019 to Aug 10, 2020 (236 days or 7 months, 23 days). In between this period I traveled out of my home country and I left her back home. We are natively Kenyans but I am currently in South Africa.
Lets now go ahead to analyze the data. I will like us to go through this in steps so that it is clear to you.
WhatsApp allows user to download the entire chat history into a text file. To do that open the conversation you are interested to get the data for. On the top right corner click the 3 dots > More > Export chat > WITHOUT MEDIA. After that a text file will download and it will contain all the messages in the conversion from the start to the end. For the sake of clarity, I saved mine as whatsapp.txt .
On opening the file whatsapp.txt you are greeted with such a file
Line 1: This is a message by WhatsApp ensuring you that the messages are end-to-end encrypted. This is a special case and we will skip it in the analysis.
Line 2 through 5: Typically, every line represents a single message from the sender. It will contain the date, time, sender and then the message.
Note: There are two senders (my fiance and I). I saved my fiance’s name as My Wife, I LOVE YOU ❤ and my WhatsApp username is koech christian.
For every single line, we need to extract the date and time the message was sent, the sender and the message itself.
Line 2 : This is an empty list to hold our data.
Line 3 and 4: Here, we are reading our file in read mode (r) and loop through each line at a time.
To understand lines 6–8, let us take one of our lines as an example:
line = 12/18/19, 08:15 - koech christian: Have woken up my dear.In 6, we will strip the line of trailing white spaces and split the line by the hyphen (-)Result: 12/18/19, 08:15 #send_timekoech christian: Have woken up my dear. #message_sectionNote: send_time still has both the date and the time while message_section has the sender and the message.Code line 7: We will split the content of message_section by the first occurrence of a full colon (:) hence maxsplit = 1. If we don't do this split function may split at the middle of the message in case a sender uses a full colon.You now know what 8 does.
Line 9 through 15: We pass our values of interest into a dictionary followed by appending it into data.
Line 19 and 20: Write data into a CSV file called messages.csv .
Exceptions raised by line 1 will be handle by try...except...
Lets load the CSV file (messages.csv) we saved in Step 3 and have a look at how a few lines look like.
Output:
The next thing is to analyze the data but before that I just want to make my guesses and probably what I expect. First, I know it is silly, but I punctuate the end of my sentences and she doesn’t LoL!. She sleeps so early even without saying good night so I bet I will be ending the conversation most days on any given day. She also wakes up earlier than I do and I will expect that she will be starting most of the “good morning” messages.
When we are back in Kenya she rarely chats on WhatsApp with me. This means that the number of messages before I traveled to South Africa (days before Feb 2, 2020) will surely be fewer than the messages we send to each other when I am here.
In the analysis of the data we will use the Pandas library for descriptive statistics and Pyplot module in Matplotlib to visualize the results of our analysis.
Lets dive in!
Out of 236 days for which we have the data, how many days did we actually exchanged WhatsApp messages?
Output:
Active days:  222First Day: 2019-12-18Last Day:  2020-08-10Difference of Dates:  236
Again let us go through what the code is actually doing
Line 1: Convert date string into Python datetime object.
Line 3 and 4 : Determine the number of active days by simply counting the number of unique values in the Date column.
Line 6 through 10: Get the first and last day for the chat history. The messages contained in the conversation are from Dec 18, 2019 through Aug 8, 2020.
Line 12–15: Get the difference between the first and the last date determined in 6–10.
From here, we can see that we have chatted on WhatsApp 222 of 236 days (94% of the time). In fact, the 14 days which makes the difference exist by choice. We used another medium of communication (when I was back in Kenya). For the time I have been in South Africa, we have 100% active days. We will discover that in a moment.
I guess I am the one with the most messages but the difference won’t be big. Lets see.
Output:
Total Messages:  34491My Wife, I LOVE YOU ❤   18030koech christian         16461
Oops! 34491 messages in total and she wrote the most of the messages. She got to hear this.
As I said, I am sure about this: she will mostly start the day and I will mostly close the day. That lady always sleep before even saying good night and always apologizes in the morning.
Morning and Night Messaging (Line 1–7)
Line 2: In this line we sort the DataFrame by Date and Time so that we bring the the first day message on top of the day chat. We also drop duplicates so that we remain with the earliest message on the DataFrame on every given day. Lastly, we count the early messages per Sender.
Line 3: Output (Morning messaging)
My Wife, I LOVE YOU ❤    157koech christian           63
Line 11: Output (Last messages of the day)
koech christian          156My Wife, I LOVE YOU ❤     64
Amazing discovery! It is true that she writes the first message most days and I do write the last message most nights. What is even surprising is how symmetric they are. It is as if we are doing our “morning runs” and “night runs” equally.
Since we started this relationship till now, I have never understood why she always sleep without saying good night. I was so uncomfortable at the start but I finally accepted her for who she is at that level.
Well, for this one I know what to expect but I don’t know how an ideal relationship should be. Should the rate of messaging deteriorate as the relationship becomes grounded or deterioration suggest a failing relationship? I don’t know but for me I think there are some factors playing. First of all, I have been working on reducing my phone addiction. This could be a cause to declining trend.
I also expect few message when I am home (Kenya) and more messages here (South Africa) because WhatsApp is the main channel of communication now.
Lets see.
It looks like my predictions were right. A rise in number of messages per day since I traveled and continued decline. To really understand this, lets plot a trend line on the plot above and also a vertical line for the day I traveled out of my home country.
There are so many methods for developing smooth trend lines. These methods includes resampling and rolling windows.
Resampling
It is often useful to resample time series data to a lower or higher frequency. Resampling to a lower frequency (downsampling) usually involves an aggregation operation — for example, computing total monthly messages from daily count.
Resampling to a higher frequency (upsampling) is less common and often involves interpolation or other data filling method — for example, interpolating daily data into hourly intervals for input to a model.
Pandas resample() method, which splits the data index into time bins and groups the data by time bin. The resample() method returns a Resampler object, similar to a pandas group-by() function. Aggregation method such as mean(), median(), sum(), etc., can then be applied to the data group for each time bin.
Rolling windows
Rolling window operations are another important transformation for time series data. Similar to downsampling, rolling windows split the data into time windows and and the data in each window is aggregated with a function such as mean(), median(), sum(), etc. However, unlike downsampling, where the time bins do not overlap and the output is at a lower frequency than the input, rolling windows overlap and “roll” along at the same frequency as the data, so the transformed time series is at the same frequency as the original time series.
For our data, we will use the rolling windows to fit the trend line
From the above Figure, we can see that the trend line (with a window size of 50 days) actually matches our prediction. Before I traveled (left of magenta vertical line), we used WhatsApp less often than after my travel (right hand side). It is also clear that the number of messages per day is declining over time.
Largest: 461 messages in a day. Date: 2020–02–13
Smallest: 1 message a day. Dates: 2019–12–22, 2020–01–01 among others
It could be very interest if we get to know the common words/emojis used by each of us. In this section we are doing exactly that.
Total Number of Words Per Sender
In this section, we will count all the words for all messages per sender (note that we are skipping some filler words like a, and, & the as shown in line 10 and 16 below)
Output:
Number of words(Myself): 105612Number of words(mylove): 113716
Again! She wins here!.
Lets go through above code snippet so that you understand what it does.
If...else... statement in line 6 and 12 is used to condition the filter on the Sender.
Line 8 and 14 ideally does the same thing. Lets see an example.
message = "Okay. Bye for now 💕💕💕💕"word_list = message.strip().lower().translate(str.maketrans('', '', string.punctuation)).split(" ")print(word_list)Result: ['okay', 'bye', 'for', 'now', '💕💕💕💕']
In this portion of code, we strip the message of trailing white spaces, change words to lower cases, strip message of punctuation marks (note that period was stripped) and lastly split the the message by white space so that we have a list of words.
What are the Most Common Words? What are the Unique Words Used Over Time?
Counter module from collections library makes things simple for us. In line 4 and 5 we take the count of words. The result is a list of word-count tuple pairs, for example,
text = "dog is a pet, hen is a bird"result = Counter(text.split()).most_common()print(result)[('is', 2), ('a', 2), ('dog', 1), ('pet,', 1), ('hen', 1), ('bird', 1)]
From here, determining the number of unique words can be done by simply getting the length of the resulting list (Line 7–9).
Output of Line 7–9:
Unique words used:Unique words She used:  8783Unique words I used:  7281
Again, she wins. I don’t know if this is a good measure of intelligence but I believe she is the smarter one. No wonder!
Next, we want to get the most common used words
Note that we already have the most common words in the two variables most_common_me and most_common_her . In the following code snippet we will just pick the top 10.
Once we have the list of most common words and the counts we can now have them on a Pandas DataFrame and also make some bar plots to visualize this as below
Results:
Honestly, data don’t lie. From her stats, I am not surprised that I, you, my and love are on top of the list. She uses them a lot and I always don’t get surprised when she writes a message like “I love you my love” LoL!. On the other hand, all I know best is how to laugh. No wonder 😂 is on top of top 10. But trust me I don’t use a whole lot of emojis I probably know only two emojis: 😂 and 🙈.
In fact, let as go ahead and analyze emoji usage.
Emojis are mostly used to express emotions and just a way to spice up communication. Lets dive in into the analysis
Line 1–4: we are using emoji package to extract emojis from sentences with the aid of the function extract_emojis , for example,
extract_emojis("K😂ip 😆rono🤣   El😂ij😁 😆ah")result: '😂😆🤣😂😁😆'
Line 6–24: Loop through all the messages and extracting all the emojis into lists with respect to sender.
Output:
Most Common emojis used (Me)('😂', 1652)('🙈', 270)('😍', 85)('😘', 52)('😋', 45)Most Common emojis used (Her)('😂', 1577)('♀', 132)('🤦', 120)('😘', 81)('❤️', 67)
Plots:
The most laughing family we are. Both of us use 😂 the most but of course I still use more than her.
It is also interesting to know distribution of chats over time.
We do talk mostly on Thursdays and Fridays but there’s no much of a difference really.
The Figure above shows the distribution of chat over a day on average. Most days between 0000H and 0500H both of us are asleep and therefore there is no much activity. Chatting gradually increases and reaches the pick between 2000H and 2100H before it also declines very fast as we go to back to sleep.
The last analysis I did was to determine the longest messages we have ever written,
Again, she leads on this with her own longest message containing 166 words while I only have 62.
In this article, I hope you had fun and also learned along the way. We cleaned, described and visualized data using pandas and by doing so you may make adjustments to make your relationship better or more generally understand string manipulation even better.
You can replicate the same using your own WhatsApp data by following the same steps.
As always, thanks for reading :-)