Discover what Singaporeans generally talk on SMS | by Jim Meng Kok | Towards Data Science

Recently, I have found a Kaggle dataset contains a corpus of English SMS messages collected for research at the Department of Computer Science at the National University of Singapore. This dataset consists of 55,835 English SMS messages taken from the corpus on March 9 2015.
By leveraging on this dataset, I am curious to find out what Singaporeans are generally talking about in their SMS conversations. Hence, this article will go through the processes of how this exercise is conducted.
In this exercise, Python programming language is used to conduct Topic Modelling, which helps to provide a quick summary of a corpus in the form of a set of topics.
# Import necessary librariesimport numpy as npimport pandas as pdimport jsonimport collections, reimport datetime# Import Gensim (LDA)import gensimimport gensim.corpora as corporafrom gensim.utils import simple_preprocessfrom gensim.models import CoherenceModel# Import NLTKimport nltkfrom nltk.corpus import stopwordsfrom nltk.tokenize import word_tokenize# For Lemmatizationfrom textblob import Word# For Visualisationimport matplotlib.pyplot as plt# For pyLDAVisimport pyLDAvisimport pyLDAvis.gensim
The above libraries would be used for data pre-processing as well as conducting topic modelling using Latent Dirichlet Allocation (LDA) from Gensim package.
LDA assumes that documents are produced from a mixture of topics. These topics generate words based on their probability distribution. Based on a given dataset, LDA backtracks and tries to find out what topics would be created for those documents in the first place (Bansal, 2016).
The dataset is in JSON format and this is imported using json.load.
# load the datasetwith open("smsCorpus_en_2015.03.09_all.json") as f:    data = json.load(f)# get the messages details only of the dataset (smsCorpus)listofMsg = data['smsCorpus']['message']# set the dictionary of messages to dataframeMsgData = pd.DataFrame(listofMsg)# getting the id and the text of the messages detailssmsMsg = MsgData[['@id','text']]# create a dataframe of the id and the text of the messages detailssmsMsg = pd.DataFrame(smsMsg)# get the number of SMS messages, which is 55,835len(smsMsg.index)
After getting the relevant data from the dataset, the first five rows of the relevant in a dataframe (smsMsg) are as shown :
The following functions would be used for pre-processing the data:
Lowercase conversion
Punctuations removal
Stop-words removal
Lemmatization
Tokenisation
smsMsg['text'] = smsMsg['text'].apply(lambda x: " ".join(str(x).lower() for x in str(x).split()))
Converting all the texts to lowercase is necessary as it is beneficial for vectorisation, the later stage of Natural Language Processing (NLP).
smsMsg['text'] = smsMsg['text'].str.replace('[^\w\s]','')
def remove_stopwords(words):    stopwords_list = stopwords.words('english')    customize_stop_words = ['lol', 'hi', 'hai', 'haha', 'hahaha', 'yeah', 'yay', 'ya', 'yup', 'yupz', 'yea', 'yep', 'yeap', 'yes', 'nah', 'nope', 'yo', 'ok', 'okay', 'xp', 'xd', 'le', 'na', 'lmao', 'im', 'u', 'youre', 'lo', 'loh', 'lor', 'la', 'lah', 'oso', 'wat', 'leh', 'mah', 'meh', 'siao', 'wah liao', 'wah lau', 'wah piang', 'kenna', 'kena', 'btw', 'kk', 'ah', 'oic', 'ahh', 'oh', 'haiz', 'omg', 'omigod', 'omygawd', 'hey', 'r', 'g', 'o', 'k', 'sibei', 'sibeh', 'jialat', 'arh', 'eh', 'xx', 'hmm', 'de', '2', 'n', 'liao', 'cant', 'thru', 'dont', 'dun', 'v', '4', 'ur', 'cos', 'coz', 'ive', 'ha', 'tt', 'h', 'b', 'th', 'brb', 'decimal', 'duno', 'huh', 'hiya', 'hm', 'ill', 'dunno', 'den', 'aa', 'wtf', 'ill', 'fucking', 'fuck', 'fk', 'shit']    tokens = word_tokenize(words)    stopwords_list.extend(customize_stop_words)    return [w for w in tokens if w not in stopwords_list]smsMsg['text'] = smsMsg['text'].apply(remove_stopwords)smsMsg['text'] = smsMsg['text'].apply(" ".join)
Punctuations are removed based on the Regular Expressions (Regex) and stop-words are removed based on the function built (remove_stopwords(words)). Stop-words usually contain determiners, prepositions, and pronouns. Psst, these are closed class when it comes to the part of speech (POS).
The stop-words removal function uses the NLTK’s stop-words list which contains the following:
There is no universal list of stop-words. Hence, in the scope of this exercise, customised stop-words were included in the stop-words removal function as well. The customised stop-words are derived from Singaporean texting slangs (E.g.: lah), vulgarities (E.g.: wtf), as well as SMS texting style of the NLTK’s stop-words (E.g.: then → den).
Removing punctuations and stop-words are necessary for the data pre-processing stage as both do not bring in value to the data but, instead, adding noise to the data.
Stemming or Lemmatization is the process of normalising the words. Instead of using stemming, lemmatization is applied in this exercise as it helps to preserve the meaning of the text messages for vectorisation, the initial stage of conducting topic modelling.
The textblob library is used for lemmatization.
smsMsg['text_after_lemmatisation'] = smsMsg['text'].apply(lambda x: " ".join([Word(word).lemmatize() for word in str(x).split()]))
Tokenization is applied on the lemmatized text into words.
words_in_docs = [nltk.word_tokenize(row) for row in smsMsg['text_after_lemmatisation']]
The initial stage of topic modelling is to create both dictionary and the corpus (vecs). Both are the two main inputs to the LDA topic model.
dictionary = gensim.corpora.Dictionary(words_in_docs)vecs = [dictionary.doc2bow(doc) for doc in words_in_docs]
In vecs, Gensim has created a unique id for each word and its word frequency in the document.
Since we have everything we need to kick-start the process of building the topic model, we need to provide the number of topics (K) as well.
In this exercise, I have set the minimum of K at 8 and the maximum of K at 20. (In case you didn’t know: In Python, the minimum value set in the range is inclusive while the maximum value set in the range is exclusive)
model_list = []coherence_values = []model_topics = []for num_topics in range(8, 21, 2):    lda_x = gensim.models.ldamulticore.LdaMulticore(corpus=vecs, id2word=dictionary, num_topics=num_topics, workers=2)    coherencemodel = CoherenceModel(model=lda_x, texts=words_in_docs, dictionary=dictionary, coherence='c_v')    model_topics.append(num_topics)    model_list.append(lda_x)    coherence_values.append(coherencemodel.get_coherence())    print("# Topics: " + str(num_topics) + " Score: " + str(coherencemodel.get_coherence()))
The variable model_list now stores 8 to 20 learned topics, based on themodel_topics collecting each num_topics.
However, we have to check the coherence values of each K from 8 to 20 inclusive, which has already stored in coherence_values, by looking at the Coherence Scores chart.
limit=21; start=8; step=2;x = range(start, limit, step)plt.plot(x, coherence_values)plt.xlabel("Number of Topics")plt.ylabel("Coherence score")plt.legend(("coherence_values"), loc='best')plt.show()
Based on the visualisation of the coherence scores from K=8 to K=20, we can infer that the optimal K is 16 due to the peak.
Evaluation of topic modelling is based on intrinsic evaluation measures — Topic Coherence and Perplexity.
Topic Coherence is applied to the top N words from the topics and it is a measure of how consistent or coherent the topics are. Whereas, Perplexity measures how well the model can predict an unseen document. A good topic model contains low perplexity and high topic coherence. The topic model built in this exercise fulfils the criteria, its result is shown below.
# Compute Perplexityprint('\nPerplexity: ', lda.log_perplexity(vecs))# Compute Coherence Scorecoherence_model_lda = CoherenceModel(model=lda, texts=words_in_docs, dictionary=dictionary, coherence='c_v')coherence_lda = coherence_model_lda.get_coherence()print('\nCoherence Score: ', coherence_lda)
To inspect these learned topics, show_topics is used. In this function, we would place 16 into num_topics and place 10 into num_words to get the top 10 words of each 16 topics.
topics = lda.show_topics(num_topics=16,formatted=False,num_words=10)for i in range(0, 16):    print(lda.print_topic(i))
Topic 0 is a represented as 0.042“time” + 0.031“think” + 0.026“bus” + 0.018“still” + 0.017“know” + 0.017“tell” + 0.016“end” + 0.015“got” + 0.014“fine” + ‘0.013“wait”.
It means the top 10 keywords that contribute to this topic are: time, think, bus, still, know, tell, end, got, fine, and wait. Furthermore, the weight of time on topic 0 is 0.042. The weights reflect how important a keyword is to that topic.
Based on the above analysis and my own interpretation, the following table shows the respective topic number and its top 10 keywords as well as my inference of the themes that each topic belongs to.
From the visualisation, first topic, second topic, and third topic are of the same size and are the largest which means the topics provide the highest importance in the entire corpus. The rest of the topics are of the same size too.
Furthermore, the top 3 most relevant terms are “thanks”, “love”, and “home” across the whole corpus as the corpus is about SMS which mainly touches on one is checking on whether his/her romantic partner has reached home.
However, the sixteenth topic is further away than the rest of the topics.
Due to the distance and closeness:
First topic (Topic 6), second topic (Topic 9), third topic (Topic 8), and eleventh topic (Topic 11) have similarities
Both third topic (Topic 8) and thirteenth topic (Topic 3) have similarities.
The reason why Topic 6, Topic 8, Topic 9, and Topic 11 are close to one another is that they mentioned about the relevance of sleep and well-wishes. As for the closeness between Topic 3 and Topic 8, both mentioned about the relevance of love and feelings.
Based on the output of the themes I have come out with, we can sense that most Singaporeans are generally talking about these through SMS:
Hanging out with friends (i.e. lunch, dinner, movie, games)
Ensuring their respective romantic partner has reached home
Requesting for help
Having a serious conversation to talk freely about their feelings or personal problems
Expressing good wishes
Informing people that they are busy, tired, or going to sleep
Notifying people that they have reached their respective place of residence
In this exercise, most parts of the process seem to be fairly subjective. These include the choice of K to start building the topic model, and human interpretation is involved when labelling the theme for each topic.
Furthermore, the constraint of building the customised stop-words is that I would have to think of the texting style of how Singaporeans text. For example, “then” is included in the NLTK’s stop-words. However, some Singaporeans typed “den” instead of “then” while texting.
Additionally, lots of fine-tuning was done in this exercise to get good results from building the topic model using LDA.
One shortcoming is that even though the data is in English, there are a few data that the texts were written in Hanyu Pinyin or Bahasa Melayu which the LDA considered them as English to derive the topics. Another shortcoming is that the actual meaning of an English word could be different from how Singaporeans use. For example, in the English dictionary, the word “got” means received. However, to some Singaporeans, “got” means have. Hence, there is a need for more comprehensive translation before it can be incorporated.
For this exercise, more could be done by adding each dominant topic for each data together with the dominant topic’s percentage contribution to the respective data. Thereafter, clustering could be conducted using the models produced by LDA. Once the process of clustering would be done, model evaluation of computing purity and entropy could be conducted to retrieve the best clustering model (high purity and low entropy). By leveraging on that clustering model, profiling could be done to identify the main theme of each cluster.
This exercise has taught us that the data pre-processing methods are very influential to the building of topic model using LDA. LDA does come into handy when it is about conducting topic modelling. Hopefully, this article has given you a rough idea of how topic modelling, one of the text mining techniques, is conducted.
Bansal, S. (2016). Beginners Guide to Topic Modeling in Python. Analytics Vidhya. August 24. Retrieved from https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/
GeeksforGeeks. (2020). Removing stop words with NLTK in Python. November 24. Retrieved from https://www.geeksforgeeks.org/removing-stop-words-nltk-python/
Tao Chen and Min-Yen Kan (2013). Creating a Live, Public Short Message Service Corpus: The NUS SMS Corpus. Language Resources and Evaluation, 47(2)(2013), pages 299–355. URL: https://link.springer.com/article/10.1007%2Fs10579-012-9197-9