Is Family Group That Bad? Results Will Shock You
By: Kaustubh Gupta
Tags: Python, Plotly, Heroku, Text Processing, WhatsApp

Text Processing, Plotly Graphs & Heroku Deployment

Is Family Group That Bad? Results Will Shock You

Who doesn't know about WhatsApp? It is the widely used mobile application irrespective of the device operating system. We all use this application to get our work done quickly and on the go. I don’t know about other countries but in India, family groups are criticized a lot due to spammy/false messages shared in the groups. This also means that a lot of Data is generated and WhatsApp gives the option to export this Data! In this article, I will show you how to dig-in this Data to discover hidden facts and finally make a deployable web app.

Photo by AARN GIRI on Unsplash

Before moving ahead, let us see how to export this data from the groups:

Image by Author (Created in Google Drawings)

Data Pre-Processing

The text file contains the timestamp of the message, the author, and the message. Data processing and manipulation can be easily done if the data is transformed into a pandas data frame and here, we will convert this textual information into this. A single entry in this file looks like this:

21/04/20, 5:47 pm - Author Name: Message sent

For every line, a simple regular expression can extract the date and time from this entry, splitting the line at this date and semicolon will give the author and message for the same but there are few problems here:

A multiline message doesn't have a new timestamp and therefore that continued message needs to be added to the main thread. There are some WhatsApp default messages like: “Messages are end-to-end encrypted” or “XYZ added you to the group” which may or may not have a timestamp and neither they are multiline. These can break our logic. Also, they are irrelevant to our analysis and can be skipped.

Considering these cases, the code for data frame extraction can be quoted as:

Abstract code to convert text file to a pandas data frame

I have not included the whole implementation here but if you are curious to know then you can head over to my GitHub repository:

Apart from converting it into a data frame, I am also interested in knowing information about the emojis shared in the group. This can be achieved by using the library named as emoji. For every message, we will check if it contains the emoji or not and then create a separate column for their count also. For the demonstration purpose, I will analyze my family group. Let’s look at how the new dataset looks:

df.tail(10)

Image by Author

This is the tail of the data frame and you can see that there are around 9k rows. The date column has a date-time data type for easier manipulation and as mentioned earlier that the data was extracted without media, the messages which contain media have been given the tag of “Media omitted” by WhatsApp. Now that our dataset is ready, it’s time to answer some questions.

Do family groups only have media chats?

All the media containing messages are renamed as omitted messages and that makes it easy to group these messages, count them and divide them by the total number of messages to get the percentage of the messages that are media. The code implementation will be:

((df[df['Message'] == ' <Media omitted> '].Message.count()) / (df.Message.count()))*100

When I ran this command on my family group dataset, it returned 53.6% that means 53 messages out of 100 are either photo, video, or any GIF. Surprisingly, when I ran the same command on my college group dataset, it returned a 3% result! That means we can say that to some extent these groups have more media but this statement is highly arguable as my college group has fewer media but other groups may have more media in the form of assignments.

Do family groups only use one type of emoji?

Emojis are everyone’s favorite thing. Sometimes people only exchange emojis while conversing to convey their emotions and talk less. It will be fun to explore how the emojis are distributed in the group. Before moving ahead, there is one thing to consider here. My native language is Hindi and when I extracted the emojis from the messages, some of the language characters are categorized as emojis. Let’s look at the top 10 frequent emojis of the group:

(Family Group Distribution) Image by Author

This is a donut plot and by looking at it I can say that the group has a lot of variety! The namaste emoji (🙏) accounts for 32% of the total emojis used. There were 20055 total emojis used out of which 498 were unique. If we look at the college group distribution, (😂) emoji leads the chart:

(College Group Distribution) Image by Author

Let’s explore some more stats to understand the nature of the groups.

Active and Lazy Members

We have the data for each day and every day from the date the group was formed. Due to this, we can analyze the activity of the members of the group over any period. Here we will consider the overall activity from the initial day. To do this, we will group the data by authors, apply count as aggregation, and plot them as a bar plot. Another method can be using the value count function to directly get the count for each author. Whatever method you choose you can plot that data and here is an example for that: