Recreating Myself from WhatsApp Chats | by Yang Chun Wei | Towards Data Science

I have always wondered if we could one day upload a copy of ourselves up on the internet. Today, although we are unable to make a high-resolution duplicate of our consciousness, I believe that I can at least capture a snapshot of my personality using present technologies. Instinctively, the usage of Artificial Intelligence comes to mind — specifically, chatbots.
There are many existing models out there for the training of chatbots. For me personally, I followed sentdex’s Youtube tutorial series which uses Python and TensorFlow. I used the nmt-chatbot (Github repository here) as the foundation and wrote a few tools to help me do the pre-processing of my WhatsApp chat data (My Github repository of tools here). There are quite a few steps which I undertook and I learned quite a bit from this experience as well.
WhatsApp is known to automatically backup your chats into Google Drive. There are existing solutions which allow you to read directly into that encrypted file, but I opted to just download the chats immediately from WhatsApp itself. To do so, you have to click on the dropdown menu inside your chat conversation with a person, click “More”, then followed by “Export Chat”.
You are then prompted with a choice to include or not to include media. For me, I chose not to as I’m unable to process anything other than text through automated means. Hopefully, in the future, I can do so in order to give the whole conversation a bit more context and improve on the accuracy of the chatbot.
After downloading the chat, open it with any text editor of your choice. You will be presented with something like this (I censored the names for the sake of privacy):
As you can see, the timestamp and the names of people are not necessary to train the chatbot. The data needs to be cleaned, so I wrote a Python script to get rid of the redundant segments.
I first removed the timestamp of the chats so that the names will define the start of each line. I wrote a script called chatparser.py, which can be found under the Chat Processing folder.
As you can see in the source below, you can add the filenames into the array so that you can process multiple text files at once. After changing my directory using the terminal, I executed the command:
$ python chatparser.py
Upon execution of the command, a new file, “WhatsApp Chat with Person Bout.txt” is generated.
Next, I separated the chats originating from me and by other people into two text files. I ran another script I wrote, chatseparator.py:
$ python chatseparator.py
This resulted in the file being generated, “WhatsApp Chat with Person Bsep.txt” and “WhatsApp Chat with Person BCW.txt” (which is me), to look like this:
The “|” character acts as a divider to indicate where the conversation is split up as a person responds or initiates a conversation. It proves to be extremely useful later on.
When texting, we tend to break up our sentences into parts instead of lumping them all together and sending it as one message. My method of classifying a question/response pair is just by detecting when the person texting changes. For every continuous chain of text typed by a person, it amounts to the sentence itself. I have to admit that this is an extremely simplified model, but it works for the majority of my chats.
Now, I execute the next two python scripts I wrote — otherstagdelete.py and mytagdelete.py — to remove the names in front of the conversation.
$ python otherstagdelete.py$ python mytagdelete.py 
I then ran my combine.py script to merge the sentences that are separated into multiple chat messages.
$ python combine.py
As we chose to import the chats without media, there are certain lines where it says <Media omitted>. To clean the data further, I just removed the question/response pairs that contain that. Do remember to remove from both documents, myside.txt and otherside.txt.
Copy all the chats originating from you to one document, and all those from the others into a separate document. By now, each question/response pair should be aligned to the same line so there should not be an issue.
If you have more than a million question/response pairs, your training results will be more accurate. With reference to the Github repository of nmt-chatbot, just put the respective training data you have created into train.from and train.to.
Upon deployment of my model, I tested it with some questions and it is evident that the chatbot has adopted my speaking style. Here are some snapshots:
Of course, it is not all beds and roses as the chatbot still spouts out nonsense for a notable portion. This is due to the fact that my training set is too small for it to be effective. However, as a proof-of-concept, I found it extremely promising.
Through this project, I think I had a couple of great takeaways which I want to share.
Firstly, a Natural Language Processing algorithm can be implemented to perhaps determine when the context of the conversation changes. This further enhances the accuracy of the question/response pair and allows for a better chatbot.
Secondly, image recognition can be also employed so that chats which include photos can also be utilized as training data. By giving images certain tags, it gives the conversation context too which is great.
AI and machine learning is really an amazing field with unlimited potential. Even if you are a beginner like me, I feel that there is much that can be done. I encourage everyone to really give learning about AI a go. I hope you found this project useful and maybe inspiring in one way or another!