Visualizing Word Embedding with PCA and t-SNE | by Ruben Winastwan | Towards Data Science

When you hear the word ‘tea’ and ‘coffee’, what would you think of these two words? Probably you will say that they both are beverages, which contain a certain amount of caffeine. The point is, we can easily recognize that these two words are associated with one another. However, when we supply the word ‘tea’ and ‘coffee’ to the computer, it can’t recognize the association between these two words like we do.
Word is not something that the computer will naturally understand. For a computer to understand the meaning behind a word, the word needs to be encoded into a numeric form. This is where the technique behind the word embedding will be useful.
Word embedding is a technique commonly applied in Natural Language Processing to transform words into numeric values in the form of vectors. These vectors will occupy the embedding space with certain dimensions.
If two words have a similar context, like the ‘tea’ and ‘coffee’ do, then the distance between these two words in embedding space will be close to one another, while the distance to other words that have a different context will be further apart.
In this article, I’m going to show you the step-by-step on how to visualize the word embedding. Since the focus of this article is not to explain in great detail about the fundamental theory behind the word embedding, you can read more about the theory in this article and this article.
To visualize the word embedding, we are going to use common dimensionality reduction techniques such as PCA and t-SNE. To map the words into their vector representations in embedding space, the pre-trained word embedding GloVe will be implemented.
Before visualizing the word embedding, normally we need to train the model first. However, word embedding is very expensive computationally to train. Hence, the pre-trained word embedding model is normally used instead. It contains the tokens and their associated vector representations in embedding space.
GloVe is a popular pre-trained word embedding model developed by Stanford researchers beside word2vec which is developed by Google. In this article, the GloVe pre-trained word embedding is implemented and you can download it here. Meanwhile, we can use Gensim library to load the pre-trained word embedding model. You can install the library using pip command as follows.
pip install gensim
As the first step, we need to convert the GloVe file format into word2vec file format. With word2vec file format, we can load the pre-trained word embedding model into the memory using Gensim library. As this file loading will take some time each time this command is called, then it will be better if we use a separate Python file just for this purpose.
Now that we have a Python file just to load the pre-trained model, next we can call it in another Python file to generate the most similar words according to input words. The input words can be any words according to your preference.
After we type the input word, the next thing would be to create a code to read it. Then, we need to specify the number of similar words based on each input word that will be generated by the model. Finally, we store the result of similar words in a list. Below is the code to do this purpose.
As an example, let’s say that we want to find out the 5 most similar words associated with ‘school’. Hence, ‘school’ would be our input word. With the code above, we would get the words ‘college’, ‘schools’, ‘elementary’, ‘students’, and ‘student’ as the result.
By now, we have the input word and the similar words generated based on it. Next, it’s time for us to visualize them in the embedding space.
With the pre-trained model, each word now can be mapped into embedding space with its vector representation. However, the word embedding comes in a very high dimensionality, which means it is impossible to visualize how words occupy the embedding space.
A method such as Principal Component Analysis (PCA) is normally implemented to reduce the dimensionality of word embedding. In short, PCA is a feature extraction technique — it combines the variables, and then it drops the least important variables while still retains the valuable parts of the variables. If you want to dig deeper into PCA, I would recommend this article.
With the PCA, we can visualize the word embedding either in 2D or 3D. Thus, let’s create the code to visualize the word embedding using the model that we have called in the code chunk above. In the code below, only 3D visualization is shown. To visualize the PCA in 2D, only minor changes should be applied. You can find the parts that you need to change in the commented sections of the code.
As an example, let’s say that we want to visualize the top 5 most similar words associated with ‘ball’, ‘school’, and ‘food’. With the code that we’ve created so far, below is the example of the visualization in 2D.
And below is the visualization of the same set of words in 3D.
From the visualization, now we can see the pattern regarding the space these set of words occupied. The words associated with ‘ball’ are placed close to each other as they have a similar context. Meanwhile, the distance between them and the words associated with ‘school’ and ‘food’ is further apart as they have a different context.
Aside from PCA, another dimensionality reduction technique that is commonly used is t-Distributed Stochastic Neighbor Embedding or t-SNE. The difference between PCA and t-SNE is the fundamental technique they both implement to reduce the dimensionality.
PCA is a linear dimension reduction method. The data in high dimensional space are mapped linearly into low dimensional space while maximizing the variance of the data. Meanwhile, t-SNE is a non-linear dimension reduction method. With t-SNE, the algorithm calculates the similarity in both high dimensional space and low dimensional space. Next, the similarity difference in both spaces is minimized using an optimization method, for example gradient descend method.
The code to visualize the word embedding with t-SNE is very similar with the one with PCA. In the code below, only 3D visualization is shown. To visualize the t-SNE in 2D, only minor changes should be applied. You can find the parts that you need to change in the commented sections of the code.
With the same example as in the PCA visualization, i.e the visualization of top-5 most similar words associated with ‘ball’, ‘school’, and ‘food’, below is the visualization result in 2D.
And below is the visualization of the same set of words in 3D.
Same as the one with PCA, note how the words that have similar context are placed close within each other while the words that have different context are placed further apart.
So far, we have successfully created a Python script to visualize the word embedding in 2D or 3D with either PCA or t-SNE. Next, we can create a Python script to build a web app for a better user experience.
The web app enables us to visualize the word embedding with a lot of functionality and interactivity. As an example, the user can type their own input words and they can also choose the top-n most similar words associated with each input word that will be returned.
The web app can be created using Dash or Streamlit. In this article, I’m going to show you how to build a simple interactive web app to visualize the word embedding with Streamlit.
First, we will use all of the Python code that we have created before and put them into one Python script. Next, we can start to create several user input parameters as follows:
Dimension reduction technique, in which the user has an option whether they want to apply PCA or t-SNE. Since there are only two options, then we could use the selectbox attribute from Streamlit.
The dimension of the visualization, in which the user has an option whether they want to visualize the word embedding in 2D or 3D. Same as before, we can use the selectbox attribute.
Input words. This is a user input parameter that asks the user to type the input words that they want, for example ‘ball’, ‘school’, and ‘food’. Hence, we can use the text_input attribute.
Top-n most similar words, in which the user needs to specify how many similar words associated with each input word that will be returned. As we can choose any number, then the slider attribute would be the best option to use.
Next, we need to take consideration of parameters that will come up in case we decide to use t-SNE. In t-SNE, there are a few parameters that we can tweak to get the best visualization result. These parameters are the perplexity, the learning rate, and the number of optimization iteration. There is no single best value for each of these parameters in each case, hence the best solution would be to let the user specify these values.
Since we are using Scikit-learn, we can refer to the documentation to find out the default value of these parameters. The default value of perplexity is 30, but we can tweak the value between 5 to 50. The default value of the learning rate is 300, but we can tweak the value between 10 to 1000. Finally, the default value of the number of iteration is 1000, but we can tweak the value as less as 250. We can use slider attribute to create these parameter values.
Now we have covered all of the necessary part to build our web app. Finally, we can wrap things up together into one whole complete script as below.
You can now run the web app using Conda prompt. Within the prompt, go to the directory of your Python script and type the following command:
$ streamlit run your_script_name.py
Next, a browser window will pop-up automatically, where you can access your web app locally. Below is the snapshot of what you can do with the web app.
And that’s it! You’ve created a simple web app with a lot of interactivity to visualize word embedding with PCA or t-SNE.
If you want to see the whole code of this word embedding visualization, you can access it on my GitHub page here.