NLP with R part 2: Training Word Embedding models and visualize results
By: Wouter Van Gils
Tags: Keras, NLP, Wordemdeddings, Umap, Word2vec

Training a GloVe word embedding

So at face value the results from Word2Vec were not mind-blowing. Some say that Word2Vec's training performs better when using larger vocabularies and hyper parameter tuning needs to be done to achieve the best results. There is an alternative on the block: Global Vectors for Word Representation. Like Word2vec , GloVe uses vector representations for words and the distance between words is related to semantic similarity. However, GloVe focuses on words co-occurrences over the entire corpus. Its embeddings relate to the probabilities that two words appear together. This is the case because GloVe is a count-based model where dimensionality reduction is done on the co-occurrence of words that appear together. Word2Vec is using a probabilistic approach where a target word and context words are predicted (and optimized based on the loss function). Many favor a GloVe model over a classic Word2Vec model because of the fact that global information is better captured, it simply works better for many tasks. So let's find out how a GloVe model is working out for our restaurant reviews.

We re-use our reviews_new dataset, which is already cleaned, as input for text2vec . As you can see terms like restaurant and eten ('food') are most common and time indications and numbers are rare.

Number of docs: 145867

0 stopwords: ...

ngram_min = 1; ngram_max = 1

Vocabulary:

term term_count doc_count

1: 0000 5 5

2: 02 5 4

3: 0300 5 5

4: 06 5 5

5: 104 5 5

---

37516: restaurant 64841 47930

37517: lekker 68548 50218

37518: bediening 78009 68484

37519: goed 88603 61218

37520: eten 111636 75699

Next we vectorize our input tokens and create a Term-Count-Matrix for GloVe to handle. We create a matrix for the word embeddings (glove_model) and execute the glove command from the text2vec package that will create the embedding matrix with (again) 32 dimension. The GloVe model learns two sets of word vectors: main and context. Best practice is to combine both the main word vectors and the context word vectors into one matrix.

Let’s find out how well GloVe is doing on our restaurant reviews. We pull up the same nearby words as we did earlier. These results look much better, a translation to English is not even required, everybody can see that pasta, spaghetti, lasagne, pizza and risotto are semantically very close. This also holds for bier ('beer'): tap, biertjes ('beers'), glas and alcoholvrij ('without alcohol') . So based on this small comparison the GloVe word embedding does a far better job than a classic Words2Vec model on our corpus. However, these results are at this point no indication that GloVe will perform better in downstream predictions tasks. To show you results of our efforts in more detail side by side we will visualize the embeddings in the next section.

pasta, pizza, lasagne, spaghetti, risotto, ravioli, zeevruchten, vongole, paddenstoelen, pizzas bier, tap, biertje, biertjes, fles, glas, flesje, wijn, frisdrank, alcoholvrij

Visualization

Up till now we’ve been looking at code, matrices en characters to see how both techniques are doing. In this section we will visualize the results of both techniques. Viewing 32 dimensions is impossible, so we need some form of data reduction. We need a method that is both fast en suitable for high dimensionality reduction. UMAP is the technique we are going to be using here, mainly because it is fast and has the ability to balance between local and global proximity very well. Another candidate for dimension reduction would be t-SNE , an excellent blog on the differences between t-SNE and UMAP can be found here.

UMAP has a lot of parameters you can play around with, the most important are n_neighbors and min_dist . As you might have read in the above blog a first thing that UMAP does is to create a high dimensional map, and the number of neighbors connected to a single point determine what that map looks like. The initial state of the map will have a focus on local detailed connections with a low value for n_neighbors and a more global big picture with a higher value of n_neighbors . Second the min_dist parameter will determine how many space you allow between points. A low value will result in close clusters at a local level, a high value will yield a more loosely connected picture.

We reduce our number of embedding dimensions (32) to just 2 so we can visualize results easily using an X and Y plot. Using 3 dimensions would also be nice, but ggplot does not allow for a 3D render in this notebook.

[1] 37520 2

We did a few iterations to find out which UMAP parameter settings gave the best results. Now for both Word2Vec and GloVe embeddings we have reduced the number of dimensions to 2. Below we attach the words that belong to rows and we create a dataframe as input for ggplot .

Our Word2Vec embedding reduced to 2 dimensions:

'data.frame': 37488 obs. of 4 variables:

$ UMAP1 : num 7.45 8.1 8.35 8.16 8.34 ...

$ UMAP2 : num 1.233 -0.677 -0.06 -0.563 0.136 ...

$ word : chr "NA" "eten" "goed" "bediening" ...

$ technique: chr "Word2Vec" "Word2Vec" "Word2Vec" "Word2Vec" ...



Our GloVe embedding reduced to 2 dimensions:

'data.frame': 37520 obs. of 4 variables:

$ UMAP1 : num -3.186 -5.385 -1.964 -0.682 1.685 ...

$ UMAP2 : num 0.252 1.316 -1.336 1.24 0.333 ...

$ word : chr "0000" "02" "0300" "06" ...

$ technique: chr "GloVe" "GloVe" "GloVe" "GloVe" ...

On the left we see the dimension reduction for the GloVe model and on the right the Word2Vec image. There are clearly more distinct spaces between local clusters for the GloVe model, the Word2Vec model looks more dense.

Let’s zoom in on the GloVe embedding some more an plot labels for all the embedding points. On the bottom left we clearly have a spot for desert related words (brownies, citroentaart 'lemon cake', spekkoek 'layer cake'). On the top right we have things that went wrong (teruggestuurd 'sent back', lauw 'lukewarm', bevroren 'frozen') and on the middle left we have a spot for snacks (bitterballen 'bitterballs', mayonaise, kipnuggets, frikandellen 'meat balls'). These embeddings clearly show that with this technique we are able to cluster words together that have the same semantic meaning.

We can also take a single word from the index for both techniques and plot semantically close words to see differences between GloVe and Word2Vec . We choose for the word pesto as it has many ingredients, can be used in various dishes and not everybody likes the taste. First we gather words that are close (based on the L2 distance), save results in a dataframe and plot results. These plots are crystal clear, the GloVe model does a far better job in understand relationship between words in our restaurant reviews.

Word embedding — wrapping it up

In this article we have explained what word embedding is and showed you how you can train your own embedding model using two different techniques. We used real world data of restaurant reviews which turned out very well for the GloVe model and not so well for the Word2Vec model. Building these embedding models required quite some decision making in order to find 'the best' model. Of course you can always avoid making all these decisions and use a pre-trained model as input for your analyses. More on the usage of pre-trained embedding models in an upcoming article.