Exploring WhatsApp Group Chat using Python
By: Srivatsan Murugan
Tags: Machine Learning, Data Science, Visualization, Naturallanguageprocessing, Python

It is no wonder that social media generates tons of information every day. Whatsapp has become one of the most popular messaging platforms where 65 billion messages are shared in a day. More than a billion people worldwide and nearly 400 million people in India use WhatsApp every day. Every message is generated in text, Analysing text messages in WhatsApp is a complex task as it involves a multitude of factors and restrictions.

Data Scientists analyze text using a process popularly called text mining which transforms the raw or unstructured data into some meaningful information that can derive insights for business and make better decisions out of it.

Now we are going to see how to analyze Whatsapp group chat in this article. This cool and fun project can help us learn a lot of interesting stuff and analyze many fun facts like :

- The number of messages sent by each person - Who has sent the most media files and emoji 😂 - Which time of the day 🕒 users were more active - Commonly used words - Which month had the maximum number of messages generated and so on.

In order to extract the group chat follow the below steps

1. Go to your WhatsApp group Click on the options — More

2.Export Chat — Without media — Send via email

A Text file will be generated.

You can get full overview of the code in this notebook

Data preparation part

Install necessary libraries

import numpy as np

import pandas as pd

import re

import seaborn as sns

import matplotlib.pyplot as plt from wordcloud import WordCloud, STOPWORDS

Python 3 always stores text strings as sequences of Unicode code points. To store those text as binary data, you must specify an encoding for that text. There are many encodings available and “utf-8” is becoming the preferred encoding for many applications.

file = open(‘WhatsApp Chat 3.0.txt’,mode=’r’)

watsup_content = file.read()

file.close()

watsup_content

mode="r” : Opens the file in read-only mode. It starts reading from the beginning of the file and is the default mode for the open() function.

At the end of every sentence “

” present in the text file. We need to split the data using split function and store it into a variable.

contents=watsup_content.split(“

”)

Here each chat is separated. Now we need to search the string using various patterns, in order to do this, we can find the pattern using regular expression and storing it to a data frame ( or ) storing the contents into a data frame and do the extraction part.

Here we stored the contents into the data frame:

df=pd.DataFrame({“data”:contents})

Once the data frame is created we are ready for the data wrangling part.

Extracting Group name from the data column

Extracting Group names

First, we are extracting group names by finding the various patterns underlying in our data.

Generally, Group name lies in :

changed the subject from “X” to “X” Person X created group “X”

Now we are going to extract group names using these 2 patterns.

df["Group_Name"]=df["data"].str.extract("(created group\s.*|changed the subject from\s.*)",expand=True)

Once extracted clean those group names by removing unwanted text.

df["Group_Name"]=df["Group_Name"].str.replace("created group ","") df["Group_Name"]=df["Group_Name"].str.replace("changed the subject from ","")

There is one problem, other than group name rows it will return Nan to all other rows in order to handle this we use ffill method. ‘ffill’ stands for ‘forward fill’ and will propagate last valid observation forward.

df[“Group_Name”].fillna(method=’ffill’,inplace=True)

Generally, the group name looks like [ changed the subject from “ Group name A ” to “ Group name B ” ]. We just need the “Group name B” as it contains the last changed group name.

First, we convert this column to “string” type and then we split the group name using “ to ”, finally extract the last element in the list. Thereby we get all the group names.

df[“Group_Name”]=df[“Group_Name”].astype(str) df["Group_Name"]=df["Group_Name"].apply(lambda x:x.split("to")) df["Group_Name"]=df["Group_Name"].apply(lambda x:x[-1])

Extracting Date from the data column

For example, each row looks like [ 18/07/19, 8:21 pm — Srivatsan: Hey hi ]

df[“Date”]=df[“data”].str.extract(‘(\d+\/\d+\/\d+)’,expand=True)

Here \d is a digit (a character in the range 0–9), and + means 1 or more times. So, \d+ is 1 or more digits.

Extracting time from the data column

Time looks like this [ 8:21 pm ]

df[“time”]=df[“data”].str.extract(‘(\d+\:\d+\s\w*)’,expand=True)

\s - space and \w - find word character

“ * ” represents 0 or more times. so it will find the word characters that have a length of 0 or more.

Extracting chat and username from the data column

df[“Name”]=df[“text”].str.extract(“(-\s\w*.*:)”,expand=True)

df["Text"]=df["text"].str.extract("(:\s\w*.*)",expand=True)

df["Word_count"]=df["Text"].apply(lambda x:len(x))

E.g: [ - Srivatsan : Hey hi ], This is how the chat and user name looks like so we extract the name part till “ : ” and after “ : ” chat part can be extracted. Once extracted the name and chat remove punctuations from respective columns using replace. Once the text column is generated create a column named word count, this will come in handy while analyzing the data.

Converting Date column into date time format

We cannot perform the other series based operations using date unless we convert into date-time format. And the next step is to convert the time column into 24 hours format so that we can easily analyze it in later stages.

df[‘Date’] = pd.to_datetime(df[‘Date’], format=’%d/%m/%y’) df["Time"] = pd.to_datetime(df['time']).dt.strftime('%H:%M')

Check if there is any emoji in chat

One of the most used things in WhatsApp is emoji. More than sending text many people tend to use emoji characters a lot to express their feelings in a better way.

df[“Emoticons”]=df[“Text”].apply(lambda x:re.findall(r’[\U0001f600-\U0001f650]’, x))

df["Emoticons_count"]=df["Text"].apply(lambda x:len(re.findall(r'[\U0001f600-\U0001f650]', x)))

Within this range of Unicode sequence, there are multiple emoji characters.

Check for any media present

Here the media files are listed as media omitted so we are going to check whether any row contains media omitted or not. The below code returns the boolean output.

df[‘Media’]=df[‘Text’].str.contains(‘<Media omitted>’)

Now the data frame is completely cleaned and all the necessary columns are extracted. Let's move on to the Analysing and visualization part.

Number of emoticons sent by each person

emoji=df.groupby([“Name”])[“Emoticons_count”].sum().sort_values() sns.barplot(x=’Name’, y=’Emoticons_count’, data=emoji,palette=”RdPu”).set_title(‘Number of emoticons sent by each person’)

plt.xticks(rotation=75)

Similarly, in the same manner, we can gather various insights out of it.

Most used Emoticons

Users tend to use lot of emoji in their chat as emojis are more engaging. It's a great way to express our feeling.

emo=df.Emoticons.apply(lambda x: pd.value_counts(x)).sum(axis = 0)

# Filtering Top 10 used emoji

emo=emo.sort_values(ascending=False).head(10)

Most used emojis

Here we can see people tend to use 😂 emoji a lot.

Total chat count from each user

Message=df.groupby(“Name”)[“Text”].count().sort_values()

Total count of words by each person

As I mentioned earlier word count columns are useful when we want to find out overall which user has used the most number of words.

Text_len=df.groupby([‘Name’])[‘Word_count’].sum()

Frequently used words

With the help of Word Cloud, we can visually represent words that are frequently used. The size of the text is based on the frequency of the particular word.

from wordcloud import WordCloud, STOPWORDS import matplotlib.pyplot as plt text = df[“Text”].values wordcloud = WordCloud( width = 3000,height = 2000,background_color = ‘white’,stopwords = STOPWORDS).generate(str(text)) fig = plt.figure( figsize = (20, 10),facecolor = ‘k’,edgecolor =‘k’)

plt.imshow(wordcloud, interpolation = ‘bilinear’)

plt.axis(‘off’)

plt.tight_layout(pad=0)

plt.show()

This is not an original chat image it’s just an example

Media shared by each person

Media=df.groupby(“Name”)[“Media”].sum().sort_values()

Amount of Media shared in each month

Series.dt.strftime() function to convert the dates in the given series object to the specified format.

E.g: Date=“2020–05–01”

df["Date"].strftime('%B %d, %Y')

output:

'May 01, 2020'

Here we need to extract only month from the date column and group by it.

month_media=df.groupby(df[‘Date’].dt.strftime(‘%B’))[‘Media’].sum()

Total Number of chats in each month

month_text=df.groupby(df[‘Date’].dt.strftime(‘%B’))[‘Text’].count().sort_values()

In which time group is more active 🕒

df[‘Time’]= pd.to_datetime(df[‘Time’])

df[“Time”].dt.hour

hour_text=df.groupby([df[“Time”].dt.hour])[“Text”].count().sort_values()

It shows that users are more active between 8 P.M to 12 P.M.

C onclusion

In this article, I have shared my approach on analyzing Whatsapp chat. Feel free to use this method and code to analyze many other insights or create more attributes based on your assumptions and goals. We can also use sentiment analysis here to find out the emotional state or view of users like positive, negative, sad, anger, joy, fear, and so on.

But sentiment analysis has its own limitations in regional Indian languages when compared to the English language. The usage of words and spelling varies from person to person and many use informal text which makes this more challenging.

Hence, Whatsapp Chat Analyzer has a number of benefits from gathering various fun facts to understanding the mental state of the participants. In other words, it has a significant place and purpose in our digital ecosystem.

Thanks for reading the article. Happy Learning!

Reach me out on Linkedin.