Build your own Whatsapp Chat Analyzer | by Samir Sheriff | Towards Data Science

Recently, after working on a number of projects from Udacity’s courses, I was on the look out for new, familiar data to analyze and what better place to start than one’s own phone.
Whatsapp claims that nearly 55 billion messages are sent each day. The average user spends 195 minutes per week on Whatsapp, and is a member of plenty of groups.
With this treasure house of data right under our very noses, it is but imperative that we embark on a mission to gain insights on the messages our phones are forced to bear witness to. This article aims to serve as a step-by-step guide to build your own whatsapp conversation analyzer, and is divided into the following 3 main topics:
Data Collection
Data Preparation
Data Exploration
Before you can get started, ensure that the following packages are installed in your Python environment (I recommend using Jupyter since you can see intermediate outputs easily while following the steps in this tutorial):
Pandas
Seaborn
Matplotlib
Jupyter (Optional but useful since you can see intermediate outputs easily while following the steps in this tutorial) OR if you are lazy like me and don’t feel like installing any of these packages, just head over to Google Colaboratory (https://colab.research.google.com) which is a free Jupyter notebook environment that comes with everything pre-installed, and get started in a jiffy!
First off, we require a whatsapp conversation to analyze. Open a whatsapp conversation you wish to analyze (preferably a group chat since they tend to be larger) and use the “Export Chat” functionality to send the entire conversation in text format to your email ID.
Important Note: When prompted by whatsapp, ensure that you do not export any media otherwise it might take ages to export.
Download the exported chat from your email inbox. It should resemble the following:
18/06/17, 22:45 - Messages to this group are now secured with end-to-end encryption. Tap for more info.25/09/16, 21:50 - Nick Fury created group "Avengers"18/06/17, 22:45 - Nick Fury added you18/06/17, 22:45 - Nick Fury added Hulk18/06/17, 22:45 - Nick Fury added Thor18/06/17, 22:45 - Nick Fury added Tony Stark18/06/17, 22:29 - Tony Stark: Here are the details for tomorrow's picnic:The park is located at 123 Main Street. Bring your own snacks, we will also be grilling. It is going to be very warm so dress appropriately. We should be getting there at noon. See you then and don't forget the sunscreen.18/06/17, 22:46 - Hulk: HULK NO CARE18/06/17, 22:46 - Hulk: HULK NO FRIEND HERE18/06/17, 22:46 - Hulk: HULK HATE LOKI18/06/17, 22:46 - Hulk: GFCHGK18/06/17, 22:47 - Thor: Stop pressing every button in there18/06/17, 22:47 - Loki: Why do you have 2 numbers, Banner?18/06/17, 22:48 - Hulk: HULK FIRST SMASH YOU THEN TELL YOU
Just like raw vegetables have to be cooked and garnished with a variety of spices to make them palatable to humans, so also this plain text file will have to be parsed and tokenized in a meaningful manner in order to be served (stored) in a Pandas dataframe:
Let us consider just a single line from the text (which we will call “raw text”) and see how we can extract relevant columns from it:
18/06/17, 22:47 - Loki: Why do you have 2 numbers, Banner?
Hereafter, whenever I wish to draw your attention to different tokens in a string s, I will present to you 2 lines.
The first line will indicate the token names enclosed within {curly braces} and their relative positions within s.
The second line will be the string s modified to indicate the actual values corresponding to each token, enclosed within curly braces.
For instance , if the string s is “Word 1, random word Word 2 :” , then I will provide the following definition:
{Token 1}, random word {Token 2} {Token 3}{Word 1}, random word {Word 2} {:)}
From this, you will be able to infer that the value of Token 1 is “Word 1”, that of Token 2 is “Word 2” and Token 3 is “:)”.
In our sample line of text, our main objective is to automatically break down the raw message into 4 tokens, and we will see how to go about this task in the next section:
{Date}, {Time} - {Author}: {Message}{18/06/17}, {22:47} - {Loki}: {Why do you have 2 numbers, Banner?}
First, in order to detect if a line of text is a new message or belongs to a multi-line message, we will have to check if that line begins with a Date and Time, for which we will need a little bit of regular expression (regex) matching (Don’t worry — it isn’t that mind boggling once you break it down, especially with some nifty tools that I’ll show you in a minute). Let us define a new method called startsWithDateTime :
The following diagram shows how regex matching detects the date and time in our message:
The following diagram gives a brief overview of all the messages detected in the sample text file:
I won’t dive into the details of how regular expressions actually work, but if you are interested, you can find more explanations about how this matching is done by visiting https://regex101.com/ and https://medium.com/tech-tajawal/regular-expressions-the-last-guide-6800283ac034 .
Coming back to our sample line, Before we ran the startsWithDateTime method, no tokens were detected in our raw sample message:
{Raw Message}{18/06/17, 22:47 - Loki: Why do you have 2 numbers, Banner?}
After we run the startsWithDateTime method, 2 tokens were detected in our processed sample message:
{Date}, {Time} -Message{18/06/17}, {22:47} - Loki: Why do you have 2 numbers, Banner?
Now that we have identified lines that contain new messages with Date and Time components, let us move to the next part of the message (everything after the hypen):
Loki: Why do you have 2 numbers, Banner?
Once again, we will require some more regular expression matching. Our objective is to detect the author of this message. While there could be a variety of patterns depending on how you have saved your friends’ names in your phone contacts app, the most commonly used patterns I have identified are as follows (Feel free to add or remove any rules as you deem fit):
Keeping these rules in mind, let us now define a method called startsWithAuthor which finds strings that match at least one of the aforementioned rules:
The following diagram shows how regex matching detects the author in our message:
The following diagram gives a brief overview of all the authors detected in the sample text file:
You can find more explanations about how this matching is done by visiting https://regex101.com/
Before we ran the startsWithAuthor method, 2 tokens had been detected in our processed sample message:
{Date}, {Time} -Message{18/06/17}, {22:47} - {Loki: Why do you have 2 numbers, Banner?}
After we run the startsWithAuthor method, 4 tokens are detected in our processed sample message.
{Date}, {Time} - {Author}: {Message}{18/06/17}, {22:47} - {Loki}: {Why do you have 2 numbers, Banner?}
Note: You might be wondering how the Message token appeared out of thin air. Well, once we have detected the Date, Time and Author tokens, what we are left with is the remaining portion of the string which is the de facto Message token
Now that we have been able to identify the Date, Time, Author and Message tokens in a single message, it is time to split each line based on the separator tokens like commas (,), hyphens(-), colons(:) and spaces( ), so that the required tokens can be extracted and saved in a dataframe. This time, let me invert things by highlighting the separator tokens instead of the Date, Time, Author and Message tokens:
Date{Comma }Time{ Hyphen }Author{Colon }Message18/06/17{, }22:47{ - }Loki{: }Why do you have 2 numbers, Banner?
Let us define a new method called getDataPoint for the task of splitting string based on the separator tokens to extract the tokens of interest:
Sample output values are shown in comments (#) beside each line.
Note: Figuring out when the value of Author token can be None is left as an exercise to the reader.
We have come to the last stage of data parsing, for which we will have to read the entire whatsapp text file, identify and extract tokens from each line and capture all data in tabular format within a list:
Initialize a pandas dataframe using the following code:
You will find all your data tabulated as shown below (Looks neat, doesn’t it?):
Finally, we have reached one of the most exciting parts of our journey — Data Exploration. It is time for us to unearth the interesting stories that all this data is trying to tell us.
Firstly, let us take a look at what pandas has to say about our data frame (df):
This command shows the number of entries (count), unique entries, most frequently occurring entries (top) and frequency of the most frequently occurring entries (freq) for each column in the data frame.
The output might look like this:
Who are the most garrulous members? Let’s take a look at the number of messages sent by the top 10 Authors in the group.
Remember how, a few sections earlier, I had given you an exercise to figure out that the author of certain messages can be None? Well, if you haven’t figured it out yet, you needn’t worry because the results of this section might give you a clue.
Let us find all those messages which have no authors, using the following code:
Do you see any pattern in the messages here?
While glancing through the original text file or the entire data frame, you might have noticed messages containing the string: “<Media omitted>”. These messages represent any pictures, videos or audio messages.
Let us find all media messages and analyze the number of media messages sent by the top 10 authors who send media messages in the group, using the following code:
Do you spot any differences between the authors who send the most messages overall, and the authors who send the most media messages?
Since we are just restricting ourselves just to the analysis of text-only messages sent by our friends in the group, let us create a new data frame (messages_df) by dropping all those messages that are either media messages or do not have an author, using the data frames obtained in the previous 2 sections:
This step could be categorized as data cleaning. Feel free to skip this step if you want to gain insights on the entire non-text data as well.
It might be interesting to count the number of letters and words used by each author in each message. So, let us add 2 new columns to the data frame called “Letter_Count” and “Word_Count”, using the following code:
This step could be categorized as data augmentation.
Now, let us describe the cleaned and augmented data frame. One important point to note here is the distinction made between columns containing continuous values vs. those containing discrete values:
Try running the describe command on the entire data frame without specifying any columns. What do you observe?
Let us take a step back and look at the overall picture. How many words and letters have been sent since the beginning of time (which in this case, happens to be since the moment the group was conceived)?
Running this code revealed that a whopping 1,029,606 letters and 183,485 words were used in the the Avengers’ group.
How many words have been sent in total by each author, since the beginning of time?
Looks like most messages contain only 1 word. I wonder what that word is!
Does it make sense to count the total number of letters sent by each author since the beginning of time as well? Well, since we already have a “Letter Count” column, I don’t see any harm in doing so. So, here goes:
Looks like most messages contain only 1 or 2 letters. Hmmm! Very interesting! Are these letters from the English language or some other symbols?
Do you know the date on which the most number of messages were sent in the history of your group? Well, fear no more, for you will find out in just a second:
22/09/17 was the most active date. Was this the date when Thanos struck, rendering everyone panic-stricken and bursting with questions? What date did you get? Do you remember anything significant happening on this date?
Do you lie awake at night wondering at what time of the day your group is most active? The truth will be revealed:
Looks like the group is mostly active at around 8:15 PM at night. Be sure to message at this time to get a quicker response.
What is the most suitable hour of the day at which to message to increase your chances of getting a response from someone?
In order to answer this question, we will have to augment the data frame to include a new column for the hour (extracted from the “Time” column), as follows:
Now, you just have to run code similar to the ones for obtaining the top dates and times:
Looks like messaging between 6 PM and 7 PM has the highest chances of eliciting responses from group members.
Congratulations! You are more knowledgeable about your whatsapp conversations now! Those were quite a lot of insights, weren’t they? What insights did you gain about your conversations? Did you find any of them useful? Did you find any other useful insights that hasn’t been covered here? Feel free to add your thoughts in the comments section so that I can improve this guide.
This analysis isn’t the be all and end all of exploration and analysis. I haven’t taken into account stuff like emojis, which are used quite a lot in conversations. Or are they? Should I discard punctuations? Why do most messages contain only 1 word?
It seems like our analysis, while answering a lot of questions, has also opened up a Pandora’s box of new questions. Well, this ain’t the end. Stay tuned for an article in the not too distant future, where I will try to dig deeper into some of these questions.
Until then, Adios Amigos and Happy Exploring! Greeting from Me...yeah ME!