Digital blindness: Learning to look at the digital conversation between two major bank brands and their audience using Python
By: Gustavo Millen
Tags: Social Media, Data Science, Python, Banking, Tech

New “digital” banks are rising in popularity in Brazil. It is hard to pinpoint the exact reasons for this, we may gain some insight on the subject by investigating data.

In this article we are going to dive deeper on tweets from two Brazilian banks: Itaú and Nubank. We will use Python to help us on this mission.

Cleaning the data

The Jupyter Notebook with the code is available here.

We start our journey with two datasets in JSON format. The data consists of tweets from both banks between 2014 and 2019 (February).

Let’s import one of the datasets and manipulate it with Pandas.

import pandas as pd

df_nubank = pd.read_json(‘nubank_tweets.json’)

df_nubank.head()

We can see that these are tweets from the bank itself, some are replies to other users. We have some interesting columns: ‘created_at’, ‘favorite_count’ and ‘retweet_count’.

Let’s sort the tweets by ‘favorite_count’ and keep tweets created after January 2018. We’ll need to clean the text, but first we duplicate it and save to a new ‘original_tweet’ column to help us later.

df_nubank = df_nubank.sort_values(by=[‘favorite_count’],ascending=False).copy()

df_nubank = df_nubank.reset_index(drop=True)

df_nubank = df_nubank[df_nubank[‘created_at’] > ‘2018–01–01’]

df_nubank[‘original_tweet’] = df_nubank[‘text’]

df_nubank.head()

Every NLP (Natural Language Processing) pipeline starts with cleaning the text. Let’s do it now!

# Lower case

df_nubank[‘text’] = df_nubank[‘text’].apply(lambda x: “ “.join(x.lower() for x in x.split()))

# Removing usernames

df_nubank[‘text’] = df_nubank[‘text’].str.replace(‘@[^\s]+’,””)

# Removing urls

df_nubank[‘text’] = df_nubank[‘text’].str.replace(‘https?:\/\/.*[\r

]*’,’’)

# Removing punctuation

df_nubank[‘text’] = df_nubank[‘text’].str.replace(‘[^\w\s]’,’’)

# Removing stopwords

from nltk.corpus import stopwords

stop = stopwords.words(‘portuguese’)

df_nubank[‘text’] = df_nubank[‘text’].apply(lambda x: “ “.join(x for x in x.split() if x not in stop))

# Removing common brazilian names -> you will need a .txt with these, it is easy to find it online

names = pd.read_csv(‘nomes.txt’, encoding=’latin’, header=None)

name_list = (names[0].str.lower()).tolist()

df_nubank[‘text’] = df_nubank[‘text’].apply(lambda x: “ “.join(x for x in x.split() if x not in name_list))

# Removing numbers

df_nubank[‘text’] = df_nubank[‘text’].str.replace(‘\d+’,’’)

# Removing small words -> e.g.: é, tá, lá, pra, etc

df_nubank[‘text’] = df_nubank[‘text’].str.replace(r’\b(\w{1,3})\b’, ‘’)

# Normalizing it

df_nubank.text = df_nubank.text.str.normalize(‘NFKD’)\

.str.encode(‘ascii’, errors=’ignore’)\

.str.decode(‘utf-8’)

# Removing empty rows

import numpy as np

df_nubank[‘text’].replace(‘’, np.nan, inplace=True)

df_nubank.dropna(subset=[‘text’], inplace=True)

# Let’s keep only the first 10k tweets from our dataframe

df_nubank = df_nubank[:10000]

df_nubank.head()

By now you can see how different our ‘text’ column looks in comparison with the ‘original_tweet’.

Let’s proceed to our first visualization: Word Clouds! We are going to use the wordcloud library.