Exploring WhatsApp data. Using whatstk library to gain insights... | by Lucas Rodés-Guirao | Towards Data Science

Analyzing data can be fascinating, fun, and can give really interesting insights about a certain topic. When this data is connected to us or our context it becomes even more fascinating to analyze it, because this information was always close and present, yet it was never looked at from an analytic perspective.
To this end, our interaction with others via social networks is a good source of data. Nowadays we use social networks and the Internet on a daily basis for work or leisure purposes. These have turned out to be really helpful, by making some processes more efficient and even easing long-distance relationships. They have impacted the way we communicate, even re-shaping how we socially behave. Social Networks register, now more than ever, how and what we communicate.
In this regard, WhatsApp has emerged as one of the most widely used Apps to chat with others [1]. In this post, I will try to show how we can easily extract and analyze data from a WhatsApp chat using python.
The outline of this post is as follows
Exporting a WhatsApp chat from your phone
Using python library whatstk
Conclusions
The process of exporting your chat from your mobile phone is fairly easy. However, it changes depending on your mobile OS (see clip below).
Make sure to select the Without Media option when exporting the chat. Once generated, you can send it to yourself via mail, and download/save it onto your computer.
On Android, several files might be exported. We are only interested in the text file (i.e. txt extension file). On iOS, the chat is exported as a zip. Once you have downloaded it onto your computer, unzip it to obtain the txt.
whatstk is a GPLv3-licensed python library, which I have helped develop, providing tools to explore, analyze, and visualize WhatsApp chats. It works with multiple chat formats and uses pandas and plotly to process and visualize the data, respectively.
At the time of writing this post, the last stable version is 0.4.1. To install it, simply use pip.
$ pip install whatstk
Requires at least Python 3.7.
We will start by loading the chat into python using class WhatsAppChat and its method from_source. For this demo, I will be using a chat that I randomly generated using library tools. An excerpt is shown below:
[2019-04-16 02:09] +1 123 456 789: Et labore proident laboris do labore ex. [2019-04-16 03:01] Mary: Reprehenderit id aute consectetur aliquip nostrud culpa, fugiat ex deserunt magna, nostrud officia id aliquip in fugiat. 🇩🇰[2019-04-17 12:56] John: Amet magna officia ullamco pariatur ipsum cupidatat, laborum sint nostrud adipiscing sit. ✈[2019-04-17 13:30] Mary: Cillum aute et cupidatat ipsum, occaecat lorem sint tempor ullamco elit. 🏊🏻[2019-04-17 15:09] John: Eiusmod irure laboris dolore anim, velit velit proident qui commodo. [2019-04-17 17:55] Mary: Aute sed quis deserunt, veniam non veniam qui ipsum velit, aliqua sunt eu incididunt incididunt duis. 🤨
In your case, change the value of parameter filepath so it points to your exported chat. The chat is automatically parsed, loaded as a DataFrame and stored as class attribute df.
Note: If you cannot load your chat this way, you may have to specify the format of your chat header manually using hformat input argument. For our chat, it would be something like:
Firstly, we will obtain some basic information on the chat activity.
We can get the timestamp of the first and last sent messages, which give us an initial time context. Next, we obtain the number of messages sent by each user as well as the volume of messages sent in each of the days that the chat was active. To accomplish this, we exploit the tools given by pandas, specifically its groupby method.
init_date = chat.df.min()
end_date = chat.df.max()
print(f"First date is: {init_date}\nLast date is: {end_date}")

First date is: 2019-04-16 02:09:00
Last date is: 2020-06-13 00:00:00

chat.df.groupby('username').agg('count')

# Day with the most number of messages sent
chat.df.groupby(chat.df.date).agg(num_messages=('message', 'count')).sort_values(by='num_messages', ascending=False)

235 rows × 1 columns
 

We observe that the recorded chat started on the 16th April 2019 and ended on the 13th June 2020. Also, seems like Giuseppe and the 1st of July 2019 were quite active!
Now, let’s take a closer look at the number of messages sent by each user. For this, we use class FigureBuilder, which eases the process of generating and displaying plotly-based visualizations.
We use cumulative counts, which means that for a given day D, we count all the interventions from each user prior to it.
Although a user sends lots of messages it does not necessarily imply that they was the one sending more characters. To find out who was the one sending the most number of characters we can use function’s argument msg_len=False.
We will now take a look at the length of the messages sent by each user since some tend to send fewer but longer messages while others send several and short messages (that is me).
Finally, the last plot illustrates how users interact with one another. In particular, it shows a matrix, which counts the number of responses sent from a user A to a user B. The library makes the assumption that a message n is always a response to the previous message n-1. While this might be far from being true it still manages to capture some sort of reality.
In this post, we have seen how we can load a WhatsApp chat in python as a DataFrame and visualize relevant insights with plotly. The visualizations have been generated using whatstk tools, but, once the chat is loaded, the possibilities are endless.
I personally find this really interesting, as I end up working with a unique, dynamic, and accessible dataset each time I explore and analyze a WhatsApp chat.
You can check the code from this post here.
July 30, 2020: Article published. whatstk version v0.3.1
May 27, 2021: Fix broken links. whatstk version v0.4.1
[1] Birgit Bucher, WhatsApp, WeChat and Facebook Messenger Apps — Global Messenger Usage, Penetration and Statistics, Messenger People 2020, blog post