Creating Word Embeddings for Out-Of-Vocabulary (OOV) words such as Singlish | by Timothy Tan | Towards Data Science

In this article, I will share how I created Singlish Word Embeddings to be used for downstream Natural Language Processing (NLP) tasks.
But why you might ask?
To understand my motivation for such a task, you first need to understand what word embeddings are and their importance in NLP.
When it comes to Natural Language Understanding (NLU), this requires a machine to comprehend the human language. To perform such a feat, the machine requires the language (written in words) to be converted into numerical values that best represents the meaning, relationship and context a particular word falls into.
These “numerical values” can be seen as Word Embeddings in NLP — words mapped to vectors of real numbers in n-dimensional space.
To put it simply, imagine a scatter plot on a 3-dimensional pane (x-axis, y-axis and z-axis). Each point on this plot represents a word in English i.e. “King”, “Queen”. You also notice that these points are pretty close (distance-wise) to each other. In addition, these points are also pretty far away from words like “Cat” or “Dog”.
To achieve such a plot, the words “King”, “Queen”, “Cat” and “Dog” had been given word embeddings (X, Y and Z numerical values) that best represent the relationship between them.
In Figure 1, we can see that words related to humans are grouped together (red) while words relating to animals are grouped together (blue). Why? Because the machine that generated these word embeddings has learned these complex relationships through multiple iterations of going through a large text corpus. It had given numeric representations to all words it has seen and trained upon.
It is like a machine saying:
“Hey! I’ve read through your text corpus multiple times over and here are the numeric values I think best represent each word!”
As a human, we need to verify such a claim yes? One way to do it is to visualise these words in vector space i.e. a scatter plot, to see where these words fall within that pane.
If the related words fall somewhat near each other as in Figure 1, then we can kind of validate that the machine has indeed learned enough to represent the words as numbers.
In short...
Word embeddings are the values that allow the machine to understand the relationships or meaning between words base on context. i.e. “King” and “Queen” were similar to one another (closer) and dissimilar (further) to words like “Cat” and “Dog”.
Of course there are more to word embeddings than the example I gave above but...
The point I want to drive across is this.
Machines work with numbers not words. Word embeddings are important because they are the numeric representations of words and this is the first step needed before conducting any downstream NLP tasks.
NLP tasks like Entity Extraction, Text Summarisation, Sentiment Analysis, Text Classification, Voice to Text, Image Captioning, Speech to Text and many more downstream tasks all require good word embeddings to get accurate results.
Making sense so far?
Now that we understand word embeddings and why they are important, let’s briefly talk about Singlish.
For my non-Singaporean readers, “Singlish” is essentially colloquial Singaporean English. It is a hybrid of English, Malay, Tamil, Mandarin and many other dialects like Hokkien, mixed and mashed together.
One of the best examples I’ve found on the internet is this.
Another of my all time favourite example of Singlish is this.
You see, Singlish isn’t just mix of different languages. Adapted from Mandarin, the use of Singlish words can change the meaning of the previous word base on the intonation of the Singlish term. i.e. “Can meh?” vs “Can bo?”
In English, the word “Can” is just that — to be able. In Singlish, that meaning changes depending on the subsequent Singlish term said with a certain intonation.
You can how see why working with Singlish is going to be a pain in NLP.
Alright then!
Now that introductions are out of the way, let’s get down to business to defeat the Huns... just kidding! (I hope you got that reference from Mulan...)
The next few sections below are the nitty gritty details on what I’ve done. It may or may not get technical but I will try my best to explain what I’ve done in simple terms.
But first, a summary.
Machines can’t understand text, but they understand numbers. Therefore, word embeddings are important as they are the numeric representations of text.It is from word embeddings that many downstream NLP tasks are performed.Singlish is a very localised language. There isn’t any out-of-the-box word embeddings for Singlish as far as I know. There is a need to create Singlish word embeddings for any future NLP works.
Machines can’t understand text, but they understand numbers. Therefore, word embeddings are important as they are the numeric representations of text.
It is from word embeddings that many downstream NLP tasks are performed.
Singlish is a very localised language. There isn’t any out-of-the-box word embeddings for Singlish as far as I know. There is a need to create Singlish word embeddings for any future NLP works.
In this section, I will first explain where I got the data from. Then I will talk about the model used to create the Singlish word embeddings. After which, I will go through my initial findings, the problems I realised from these findings, how I rectified the problem before finally revealing my finalised result from this.
The data was scrapped from our all time favourite Singapore forum called “Hardware Zone” ( https://forums.hardwarezone.com.sg/). It was a forum initially made to be IT-oriented but as with all forums, it has deviated to being a place many Singaporeans go to to talk about anything and everything under the sun.
The best part?
They mostly use Singlish in the forum. Perfect for any deep learning task. Here is an example of how a thread comment would look like.
['got high ses lifestyle also no use treat chw like a dog if me i sure vote her she low ses man take people breakfast ask people buy ckt already dw pay up tsk tsk tsk\n', '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   children big liao no need scare tio pok\n', 'she low ses man take people breakfast ask people buy ckt already dw pay up tsk tsk tsk i dont get her thinking actually if she is rich she could even fly over to kl to eat buffet and then fly back in 1 hour dunno why want to save over this kind of small thing to ppl like her like food\n']
It may seem like just bad and broken English but actually, to a Singaporean reading this, we understand all the implicit meanings from the slangs and English used here.
Data collection-wise, 46 main threads were scrapped with a total of 7,471,930 entries. Below is the breakdown of number of threads by entry.
Eat-Drink-Man-Woman                                    1640025Travel and Accommodation                                688253Mobile Communication Technology                         582447Mass Order Corner                                       574927MovieMania                                              451531Gaming Arena                                            426976Campus Zone                                             322146General Merchandise Bazaar                              286604Money Mind                                              264523Internet Bandwidth & Networking Clinic                  243543Music SiG                                               227198Hobby Lovers                                            224994Headphones, Earphones and Portable Media Devices        206141Notebook Clinic                                         184116HomeSeekers and HomeMakers                              153683Apple Clinic                                            139810Hardware Clinic                                         133421Cars & Cars                                             115286The Tablet Den                                          108793Fashion & Grooming                                       98211Electronics Bazaar                                       75192Degree Programs and Courses                              49734Football and Sports Arena                                47769Software Clinic                                          43553Health & Fitness Corner                                  21665The "Makan" Zone                                         21358National Service Knowledge-Base                          16545Current Affairs Lounge                                   16259Home Theatre & Audiophiles                               14888The House of Displays                                    13987Employment Office                                        12146Other Academic Concerns                                   9927Tech Show Central (IT Show 2018)                          9895Parenting, Kids & Early Learning                          9756Pets Inner Circle                                         6525Wearable Gadgets and IoT                                  5735The Book Nook                                             5238Digital Cameras & Photography                             4714Ratings Board                                             3858IT Garage Sales                                           3549Diploma Programs and Courses                              2348Certified Systems, IT Security and Network Training       1798Post-Degree Programs & Courses                            1724Online Services & Basic Membership Support/Feedback        876Design & Visual Art Gallery SiG                            246HardwareZone.com Reviews Lab (online publication)           18
Having manually reviewed the data quality from each thread, some threads like “Travel and Accommodation” were used for mainly sales and thus, rejected from the analysis.
I ended up only using data from “MovieMania” which did not include any form of advertisements nor sales within the thread. I also deemed 451,532 entries a large enough set to train decent word embeddings.
The average length of each entry in this thread was 27.08 words. This gave me a corpus of 12,675,524 words. In terms of number of unique vocabulary words used, the number came down to 201,766. I kept the top 50,000 words based on frequency and labelled the remainder with an “unknown” token for training.
There is no point keeping infrequent words for training. If the word only appears a couple of times, there is no learning that can take place. Hence, the reason why I replaced all infrequent words with an“unknown” token.
I also cleaned the data by simply removing all forms of punctuation and standardising the cases by lower casing them.
Technical Summary:
No. of words: 12,675,524No. of sentences: 427,735Avg. No. words per sentence: 27.05No. of unique vocabulary words used: 50,000
Now that the data is out of the way, the next segment will talk about the actual model used to train word embeddings.
The skip-gram model is an unsupervised machine learning technique that is used to find the most relevant words for a given word.
Take this phrase for example,
The skip-gram model tries to predict the context words given an input word. In this case, given the word “Fox”, predict “quick”, “brown”, “jumps” and “over”.
Now imagine the model scanning through my entire training corpus from Hardware Zone. For each sentence (427,735 sentences), loop through each word, extract out it’s context words and use the input word to predict those context words.
An error function is also calculated per sentence. The goal of the model is to minimise this error function through multiple iterations by slowly adapting the weights within the model.
I iterate this “learning” until the error function starts to stabilise before extracting the weights with the model as my word embeddings.
What I attempted to explain above, is essentially how neural networks work. Architecturally, it will look something like this:
Following so far?
Great! Before I get to the results, let’s review some technical specifications in the model building process.
Technical Specifications:Context size = 3Learning Rate = 0.025Learning Rate Decay = 0.001No. of Epochs = 25No. of word dimensions = 100No. of Negative Samples = 3Total Training Time = 17 hrs 05 mins (~= 41 mins per Epoch)
For those who are wondering what these technical jargon refers to, context size refers to the number of words to predict i.e. “quick”, “brown”, “jumps”, “over” was a context size of 2 — Two words before the input word, “Fox” and two words after.
Learning rate and decay refers to how the model adapts it weights over the iterations. i.e. how much it learns per iteration.
Epoch refers to how many iterations of the entire training set to loop through. For example, I had 427,735 sentences in my training set, I am looping through this 25 times.
Word dimensions refers to the number of numerical values to best represent a word. In my example in the introduction, I used 3 dimensions to keep it simple and easy to understand. In actual fact, we can actually go up to 300 dimensions.
Negative sample size is a parameter that comes from Negative sampling — a technique used to train machine learning models that generally have more negative observations compared to positive ones. Recall I kept 50,000 vocabulary words to use.
If I were to use “Fox” to predict “Quick”, there will only be 1 correct answer, versus 49,999 wrong answers. The probability to get the prediction of “Quick” correct would be insanely low.
Hence to speed up the learning process, negative sampling is used to reduce to number of negative levels. i.e. Instead of looking at 49,999 wrong answers, I only look at 3 random wrong answers. Hence, No. of Negative Samples = 3.
Still with me yeah? Fantastic! Let’s carry on!
Here are the results of the first run of the trying to generate word embeddings for Singlish.
What you see below are similarity scores (0 to 1) between word-pairs and the top 10 words that tend to be closest to the word of interest. The higher the number, the more similar the words are in terms of context.
As an initial test before even looking into Singlish words, I decided to look at the scores for words like “her” vs “she” and “his” vs “he”. If these scores were not high, I would have deemed that the model has not trained well enough.
Thankfully, the scores are pretty high and decent.
Similarity between 'her' & 'she': 0.9317842308059494Closest 10:her 0.9999999999999994she 0.9317842308059496who 0.8088322989667506face 0.7685887293574792and 0.731550465085091hair 0.7196624736651458shes 0.7191209881379563when 0.7119862209278394his 0.7107795929496181that 0.7091856776526962********************************************************Similarity between 'his' & 'he': 0.897577672968669Closest 10:his 1.0he 0.8975776729686689him 0.8446763202218628who 0.775987111217783was 0.7667867138663951that 0.7528368024154157father 0.749632881268601son 0.7281268393201477become 0.7264880215455141wife 0.711578758349141********************************************************Similarity between 'jialat' & 'unlucky': 0.011948430628978856Closest 10:jialat 1.0sia 0.8384455155248727riao 0.8266230148176981liao 0.8242816925791344sibei 0.814415592977946hahaha 0.8064565592682809ya 0.8045512611232027meh 0.7954521439129846lol 0.7936809689607456leh 0.7920613014175707********************************************************Similarity between 'jialat' & 'bad': 0.6371130561508843Closest 10:bad 1.0quite 0.8823291887959687good 0.8762035559199239really 0.8758630577100476very 0.8731856141554037like 0.8728014312651295too 0.8656864898051815damn 0.8599010325212141so 0.8486273610657793actually 0.8392110977957886********************************************************Similarity between 'bodoh' & 'stupid': 0.1524869239423864Closest 10:bodoh 0.9999999999999998628 0.4021945681425326u4e3au56fdu5148u75af 0.3993291424102916beck 0.39461861903538475recieve 0.39110839516564666otto 0.3839416132228821gaki 0.34783948936473097fapppppp 0.3418846453140858bentley 0.3344963328126833hagoromo 0.3331640207541007********************************************************Similarity between 'bah' & 'ba': 0.5447425470420932Closest 10:bah 0.9999999999999998lei 0.7051290703273838nowadays 0.698360482336586dun 0.6968374466521237alot 0.6767383433113785type 0.6745085658120278cos 0.6711909808612231wat 0.6682283480973521ppl 0.6675756452507112lah 0.6671682261049516********************************************************Similarity between 'lah' & 'la': 0.8876189066755961Closest 10:lah 1.0meh 0.8877331822636787la 0.8876189066755962dun 0.8865821519839381mah 0.8793885175949425leh 0.8723455556110296cannot 0.8686775338961492ya 0.8661596706378043u 0.8549447964449902wat 0.8542029625856831********************************************************Similarity between 'lah' & 'leh': 0.8723455556110294********************************************************Similarity between 'lah' & 'ba': 0.6857482674200363********************************************************Similarity between 'lah' & 'lor': 0.8447135421839688********************************************************Similarity between 'lah' & 'hor': 0.722923046216034********************************************************Similarity between 'lor' & 'hor': 0.6876132025458188Closest 10:lor 1.0u 0.8925715547690672cannot 0.865412324252327dun 0.8509787619825337leh 0.8508639376357423lah 0.8447135421839689ya 0.8438741042009468la 0.8403252240817168meh 0.8356571743730847mah 0.8314177487183335********************************************************Similarity between 'walau' & 'walao': 0.4186234210208167Closest 10:walau 1.0nv 0.6617041802872807pple 0.6285030787914123nb 0.6248358788358526la 0.6207062961324734knn 0.6206045544509986lah 0.6158839994483083lo 0.6102554356797499jialat 0.6079250154571741sibei 0.6076622192051193********************************************************Similarity between 'makan' & 'eat': 0.6577461668802116Closest 10:makan 1.0jiak 0.7007467882204779go 0.6911439090088933pple 0.65857421561786eat 0.6577461668802115food 0.6575154915623017kr 0.6545185140294344sg 0.6473315303433985heng 0.6422265572697313beo 0.6354614594882941********************************************************Similarity between 'makan' & 'food': 0.6575154915623018********************************************************Similarity between 'tw' & 'sg': 0.7339666801539345Closest 10:tw 0.9999999999999997tiong 0.7723149794376185sg 0.7339666801539344lidat 0.7330705496009475hk 0.7258329008490501taiwan 0.7195021043855226tiongland 0.7171170137971364pple 0.7130953678674011yr 0.7017495747955986mediacorpse 0.6954931933921777********************************************************Similarity between 'kr' & 'sg': 0.7889688950703608Closest 10:sg 1.0go 0.7940974860875252kr 0.7889688950703608tiongland 0.7675846859894958ongware 0.7674045119824121yr 0.7584581119582794pple 0.7536492976456339time 0.7533848231714694buy 0.751509500730294tix 0.743339654154326********************************************************
There are some interesting findings from the above results that I’d like to point out.
Overall, I would say the model has learned and performed decently well on understanding Singlish words.
Not the best, but decent.
It’s no surprise that the model has learned that “Makan” (Malay for “Eat”), “Food” and “Eat” were found to be similar. But what’s interesting is the word “Jiak” — dialect for “Eat” and commonly used in Singlish sentences, was picked up and deemed similar to “Food”, “Makan” and “Eat” as well.
This looks to me that the model has indeed learned some Singlish words well.
The next interesting result comes from the abbreviations of country names. As humans reading the result, we know “sg” refers to “Singapore”, “kr” — “Korean”, “tw” — “Taiwan” & “hk” — “Hong Kong”. To a machine however, it is not that simple, the model did not have prior knowledge of these. Yet, it has managed to group these terms together.
Very interesting indeed!
This is another validation point that tells me the model has trained decently well on Singlish words.
Now, instead of just staring at numbers, there is a method that helps us visualise words in vector space. This is called T-distributed Stochastic Neighbor Embedding (TSNE).
TSNE can be loosely seen as a dimensionality reduction technique to visualise high-dimensional data. i.e. word embeddings of 100 dimensions. The idea is to embed high-dimensional points in low dimensions in a way that respects similarities between points.
In English, convert 100 dimensions into 2 or 3 dimensions and retain the embedded information as much as possible.
If I were to run TSNE (n=2 dimensions) on some of my results above and display it on a scatter plot, it will look like this:
Not too shabby a result.
While the results above looked promising, there were some issues that needed addressing.
I only realised this when I was reviewing the results. The following section will explain further.
First off, as you can see, Singlish words like “lah”, “lor”, “meh” seem to be similar to each other but...
So what? What does it actually mean?!
After thinking about it, it is not exactly useful.
Singlish phrases usually appear as bigrams (2 words) that really encapsulates the meaning of the context in which it was used. A good example is Figure 3 — The power of “Can”. The bigrams “can meh?” vs “can lah!” have very different meanings.
If we were to look at the words independently, “can” and “meh”, that meaning is lost.
Therefore, having a vector representation of the words “meh” vs “lah”alone is not entirely useful here.
To remedy this issue, I needed a way to keep these words together. Skip-Gram trains on unigrams, yes. But what if I could convert these bigrams to unigrams first? i.e. “Can meh” -> “Can_meh” *note the underscore.
The second problem is not as serious as the first, but worth some air time.
It is the inherent problem with dimensionality reduction techniques like TSNE. You see, information loss by downsizing from 100 dimensions to 2 dimensions is inevitable. As much as I love visuals, I realised that I could only get the true understanding of the results by looking at numbers and not the visual.
Nothing to solve here.
Just food for thought.
Now let’s take a look at my attempt to solve the first problem.
After realising the issue with Singlish, I needed a way to convert my bigrams to unigram and yet keep the information or meaning of those words.
The way I could do that was by calculating the PMI of all possible bigrams in the corpus.
For example, the formula in Figure 7 reads, take the log of the probability of the occurrence of pair of words (i.e. “can meh”), divided by the probabilities of the occurrences of each individual words.
The breakdown of my process is as such:
If x = “can” and y = “meh”,
Count the number of times “can meh” appears together.Count the number of times “can” appears alone.Count the number of times “meh” appears alone.Apply PMI and get a score.Set some threshold parameter. If score is above threshold convert all occurrences of “can meh” into “can_meh” i.e. bigram to unigram.Retrain the entire skip-gram model.
Count the number of times “can meh” appears together.
Count the number of times “can” appears alone.
Count the number of times “meh” appears alone.
Apply PMI and get a score.
Set some threshold parameter. If score is above threshold convert all occurrences of “can meh” into “can_meh” i.e. bigram to unigram.
Retrain the entire skip-gram model.
The steps from 1 to 5 is actually known as building a phrase model.
And so... the quest to retrain my 17 hour model ensued...
I calculated the PMI scores for all possible bigrams and set a threshold variable. I actually used the normalised version of the PMI formula above for calculation, but the concept is the same.
Technical Specifications of Normalised PMI phrase model:
Minimum count of word = 5Threshold = 0.5 (range from -1 to 1)
Minimum count of word = 5
Threshold = 0.5 (range from -1 to 1)
['as we are left with just 2 days of 2017 hereu2019s a song for everyone here edit to weewee especially lol', 'good_morning click here to download the hardwarezone_forums app']['wah_piang . why every time lidat one . siao liao can meh . wah_lau . can lah . can la . of course medicated_oil . can anot']
As you can see, the threshold affects how many bigrams gets converted to unigrams. I did not manage to get bigrams of “can meh” to be converted to unigrams because if I were to set the threshold any lower, many other non-bigram related words will start to be converted into unigrams.
Sadly, there is a trade-off to make here.
After accounting for the bigrams, I retrained the model (which now took me 18 hours to train) and received the results below.
Notice how many non-related bigrams became unigrams? i.e. “for_her”, “with_her”. This is what I meant as trade-off in the phrase modelling step.
Because of all these non-related bigram, a number of the scores have fallen.
Similarity between 'her' & 'she': 0.8960039000509769Closest 10:her 1.0she 0.8960039000509765when_she 0.7756011613106286she_is 0.7612506774261273with_her 0.7449142184510621who 0.7348657494449988for_her 0.7306419631887822shes 0.7279985577059225face 0.7192153872317455look_like 0.718696491400789********************************************************Similarity between 'his' & 'he': 0.8129410509281912Closest 10:his 1.0he 0.8129410509281914him 0.804565895231623he_was 0.7816885610401878when_he 0.7747444761501758that_he 0.7724774086496818in_the 0.7622871291423432himself 0.7611962890490288was 0.7492507482663726with_his 0.7241458127853628********************************************************Similarity between 'jialat' & 'unlucky': 0.1258840952579276Closest 10:jialat 0.9999999999999996den 0.7104029620075444liao 0.7050826631244886chiu 0.6967369805196841heng 0.686838291277863hahaha 0.6860075650732084riao 0.6810071447192776la 0.6804775540889827le 0.676416467456822can_go 0.6756005150502169********************************************************Similarity between 'jialat' & 'bad': 0.4089547762801133Closest 10:bad 1.0but 0.8388423345999712good 0.8244038298175955really 0.8192112848219635i_think 0.7970842555698856very 0.7965053100959192like 0.785966214367795feel 0.783452344516318too 0.7788218726013071i_feel 0.7721110713307375********************************************************Similarity between 'bah' & 'ba': 0.5793487566624044Closest 10:bah 1.0say 0.6595798529633369lah 0.6451347547752344dun 0.6449884617104611sure 0.627629971037843bo_bian 0.6251418244527653this_kind 0.6223631439973716ppl 0.6196652346594724coz 0.61880214034487mah 0.6146197262236697********************************************************Similarity between 'lah' & 'la': 0.8863959901557994Closest 10:lah 0.9999999999999998la 0.8863959901557996meh 0.8739974021915318mah 0.8641084304399245say 0.8589606487232055lor 0.8535623035418399u 0.8304372234546418leh 0.8275930224011575loh 0.8189639505064721like_that 0.8170752533330873********************************************************Similarity between 'lah' & 'leh': 0.8275930224011575********************************************************Similarity between 'lah' & 'ba': 0.7298744482101323********************************************************Similarity between 'lah' & 'lor': 0.8535623035418399********************************************************Similarity between 'lah' & 'hor': 0.7787062026673155********************************************************Similarity between 'lor' & 'hor': 0.7283051932769404Closest 10:lor 1.0u 0.8706158009584564lah 0.8535623035418399mah 0.8300722350347082meh 0.8271088070439694or_not 0.8212467976061046can 0.8202962002027998la 0.8136970310629428say 0.8119961813856317no_need 0.8113510524754526********************************************************Similarity between 'walau' & 'walao': 0.37910878579551655Closest 10:walau 0.9999999999999998liao_lor 0.595008714889509last_time 0.5514972888957848riao 0.5508554544237825no_wonder 0.5449206686992254bttorn_wrote 0.5434895860379704hayley 0.5418542538935931y 0.5415132654837992meh 0.5397241464489063dunno 0.5377254112246059********************************************************Similarity between 'makan' & 'eat': 0.6600454451976521Closest 10:makan 0.9999999999999998go 0.7208286785168599den 0.697169918153346pple 0.6897361126052199got_pple 0.6756029930429751must 0.6677701662661563somemore 0.6675667631139202eat 0.6600454451976518lo 0.6524129092703767there 0.649481411415345********************************************************Similarity between 'makan' & 'food': 0.4861726118637594********************************************************Similarity between 'tw' & 'sg': 0.6679227949310249Closest 10:tw 1.0tiong 0.7219240075555984taiwan 0.7034844664255471hk 0.7028979577505058tiongland 0.6828634228902605mediacock 0.6751283466015426last_time 0.6732280724879228sg 0.6679227949310248this_yr 0.6384487035493662every_year 0.6284237559586139********************************************************Similarity between 'kr' & 'sg': 0.7700608852122885Closest 10:sg 1.0in_sg 0.8465612812762291kr 0.7700608852122885pple 0.7599772116504516laio 0.7520684090232901go 0.7509260306136896sure 0.7156635106866068come 0.7116443785982034tiongland 0.7015439847091592ongware_wrote 0.6974645318958415********************************************************Closest 10:wah_piang 1.0tsm 0.4967581460786865scourge_wrote 0.4923443232403448aikiboy_wrote 0.4887014894882954sian 0.4871567208941815ah_ma 0.48058368153798403lms 0.4790804214522433ruien 0.47420796750340777xln 0.46973552365710514myolie 0.4682823729806439********************************************************
In short, the results above was worst off than the previous results.
I think that the this issue came from the lack of data. If I had more examples of people using “can meh” or “can lah”, the phrase model would have picked up these bigrams without having me set a low threshold. 0.5 threshold is really low in my opinion.
I caught too many false positives.
Rubbish in, rubbish out right? This was exactly what happened here.
As you can imagine, I still have ways to go in tuning my model to get decent Singlish word embeddings for my future works.
What I would do differently is this:
Recode my scripts to run distributed. Or better yet, code to run in a GPU. This will allow me to run on a much larger corpus i.e. the whole of Hardware Zone and not just one thread of it.Train a better phrase model. i.e. the PMI portion.
Recode my scripts to run distributed. Or better yet, code to run in a GPU. This will allow me to run on a much larger corpus i.e. the whole of Hardware Zone and not just one thread of it.
Train a better phrase model. i.e. the PMI portion.
For now, I will keep the word embeddings for the first model.
Edit: If you’re interested in finding out about how to deal with misspellings, abbreviations or other OOV words in your corpus, I’ve written another article about it here!
I’ve definitely learned a lot from this exercise and I hope that by me sharing my methodologies, thought processes and results, you’d be able to do the same in your respective local languages.
With that, adios amigos!
Hope that you’ve enjoyed reading this as much as I’ve enjoyed doing it!
Feel free to share the article if it helps anyone out there!
If you are looking for NLP datasets to work with, click here for a curated list I created! :)
LinkedIn Profile: Timothy Tan