Generate Meaningful Word Clouds in Python | by Bryan Dickinson | Towards Data Science

I’m sure you’ve seen a word cloud. Generally, it’s a photo filled with words where the size depends on the frequency of times the word appears in the text you’d like to analyze. I wasn’t privy to this, but apparently, there has been/is (not sure) some pretty strong feelings against/for word clouds. I have read that they are bad, lame, equating them to a modern pie chart and even comparing them as the new mullet. Ouch. A mullet? Come on.
I’ve also read some ways to improve word clouds and useful times to use them. Regardless of which camp you are in, I found that leveraging a compelling graphic or visualization in a presentation, engages your audience, prompts a reaction, can start a conversation, can be influential, and opens the door for more detailed analysis. A visually compelling word cloud art can draw readers in. With that said, I won’t get further into the pros and cons of word clouds, you check out the above links for that.
We will use a couple of different methods to extract some meaningful words out of the text. We will then generate some word clouds using the Python libraries WordCloud, pandas, and NumPy.
WordCloud is a word cloud generator in Python. You can install WordCloud by one of the following commands.pip install wordcloudconda install -c conda-forge wordcloudCheck out installation details here, and you can read through the WordCloud documentation here.
The data we will be using is the democratic primary debates for the 2020 presidency. You can find the full dataset here via Kaggle. The data has been cleaned and filtered. The Github repo has the cleaning steps included.
1| Import Libraries and Data
Before creating a word cloud the text stopwords should be updated specifically to the domain of the text. For example, if we were completing word clouds from customer tweets for an airline company, we would probably get words like ‘plane’, ‘fly’, ‘travel’ and they may not be of any significance to any analysis you are completing.
In this dataset, additional stopwords were included because they appeared a lot in the text but did not contribute to the analysis. For example words like ‘senator’, ‘congressman’, ‘people’, ‘fact’ were all words that were used by many candidates in sentences like ‘I agree with Senator Sanders...’ or ‘The fact is...’, ‘The American people want..’ and did not necessarily contribute to words of meaning for the word cloud.
2.1| Method 1 — Use WordCloud to ProcessThe simplest and fastest way to create a word cloud is to simply use WordCloud to process the text.
The text needs to be in one long string in order for WordCloud to process it. We filter the data to ‘biden’, create a list of his responses, and join the list to create one long string of text. We then create the word cloud object, use the generate() method, and pass our string of text. Lastly, we use plt.imshow to display the image.
Let's take a look at the parameters from the documentation:stopwords: this parameter takes in a set of strings (words) that will be eliminated from the word cloud. We used our updated list of stopwords here.collocations: This parameter takes a bool statement, and will generate bigrams from your text if set to ‘True’ We don’t actually see any bigrams here.background_color: sets the background color, default is black
The result ...looks a bit like gibberish and doesn’t look too informative. Words ‘re’, ‘said’, ‘make’, and ‘said’ seem to be the most frequent words. Those don’t seem to be too significant in Joe Biden’s message whereas words like ‘cost’, ‘medicare’, and ‘world’ in the background. An option could be to continue updating stop words. A challenge with that process is knowing when to stop. When do you know you have a good amount of words without removing useful words? I’m not dismissing this method, I’ve used it before.
We can use the process_text() and words_ methods to display the word count and relative counts from the text respectively.
# output[('re', 148), ('make', 117), ('ve', 106), ('said', 95), ('able', 88)][('re', 1.0), ('make', 0.7905405405405406), ('ve', 0.7162162162162162), ('said', 0.6418918918918919), ('able', 0.5945945945945946)]
You can take a peek at other candidates and you’ll notice there a similar result of non-meaningful words appearing high on the list. Let’s tweak some additional parameters of WordCloud to improve the words shown. min_word_length: is the minimum number of letters a word must have to be included in the word cloudcollocation_threshold: is a Dunning likelihood score. Bigrams in the text must reach a score greater than this parameter to be counted as a bigram. The default is set at 30.
We’ll add a min word length of 4 and a collocation_threshold of 3 to include more bigrams. Lastly, and perhaps most importantly, I will be utilizing a different set of stopwords. The original set was imported from WordCloud. I will import and use the stopwords offered in the SpaCy library and update that list with my own. No real reason for using SpaCy’s list other than I’ve used it in the past and have gotten good results. There are many different stop word libraries you can use.
We see a pretty good improvement! We are getting some different words, including bigrams like ‘Donald Trump’, ‘Barack Obama’, ‘public option’, and ‘middle class’. Some more tweaking/updating of stopwords might improve this.
2.2| Method 2— Utilizing Word FrequenciesThe previous method used a string of text. What if you don’t necessarily have access to the full text or want to use word frequencies directly? A WordCloud alternative to the generate() method is the generate_from_frequencies() method that will take a dictionary of words and their frequencies and create a word cloud from the counts. Let's give it a try.
We still have the full text, so we will utilize CountVectorizer to create a matrix of word counts. If you already have a dictionary of counts or a bag of words matrix, you can skip this step.
Now we just need to extract one row of this dataframe, create a dictionary, and place it into the WordCloud object.
The new word cloud looks somewhat similar to the previous version. There are similar top frequent words with some differences. When the generate_from_frequencies method is used, it ignores some of the parameters including the collocations and stopword parameters. CountVectorizer processes the text, including stop words and lemmatization. If we wanted to make additional tweaks to the words, it would need to be done prior to placing them into the WordCloud object. We can see that ‘Donald’ is lemmatized to ‘Don’ and that there are no bigrams in this version.
Another option to creating a word cloud from frequencies is to utilize the collections Counter object to create a dictionary that you can utilize with WordCloud. This would require preprocessing your text including tokenizing the words, then using the counter object to count each word. Given that you need the whole text it would be easier to utilize the full text with the WordCloud object to perform this for you.
2.3| Method 3— Log Odds RatioFrom the last two word clouds, we got pretty good groups of words that encompass what these candidates have said. If you look at top words from multiple candidates, you start to see that similar words start showing up for all — but what about words or phrases that are specific to a certain person? For example, if you’ve watched these debates you may have noticed that Amy Klobuchar mentions ‘lead democrat’ and Biden likes to count off his points (‘number one’, ‘number two’ ..) quite often. We could get a view of important words or phrases that are mentioned by a particular candidate, but not others.
Going back to our analyzing customer tweets for airplane company example. We may want to capture what segment of our customers are mentioning. What do our business travelers think vs other travelers? Many words may be the same such as ‘seat’ and ‘experience’, but they may have different thoughts and priorities than say urgent travelers who also mention words such as ‘seat’ and ‘experience’. Differentiating these words can help get a better understanding of these different segments of customers.
Put simply we want to see which words are most common from a specific candidate, say, Pete Buttigieg, relative to all other candidates. And we’ll do this with each candidate.
To do this we’ll use the measure of log odds ratio calculated for each word as:
We used the bag of words dataframe and transformed each row using the calculation above. The result is a data frame showing the log odds of each word being said by a particular candidate.
We tried to identify words that were most unique to each candidate. Here are the top 10 words for four candidates can you match them to the correct candidate?
Following the same steps above for generating a word cloud via word frequencies, we can now create a new word cloud.
The great thing about this method is that n-grams are generated using this method. We see Biden’s message of restoring the soul of America and his mentions of the Violence Against Women Act. There are other methods that can be used, such as weighted log-odds & tf-idf.
So which text do we use? With the premise of creating stunning visual vs analytical analysis with words. I combine both to give a general sense of frequent words and common words from and to that candidate.
2.4| Combine DictionariesWe have two different dictionaries/word frequencies (methods 1 & 3) that we can utilize separately or combine to create an all-encompassing word cloud. After combining them, we will make one more tweak. You’ll notice that a word may be spelled differently. Like healthcare and health care. We will add these together to give a better representation of the word.
This may look a little crazy. This code is setting the second dictionary’s most common word’s frequency equal to the first dictionary’s most common word’s frequency. We now combine the dictionaries and combine ‘healthcare’ and ‘health care’ into one key for a better representation. The top 5 entries and word cloud are displayed below.
[('number number', 56), ('argued', 44), ('special forces', 41), ('single solitary', 41), ('restore soul', 40)]
Great, we see a blending of both worlds that were very frequent of the candidate, and words that are common to that candidate alone. We have our text next up will be creating our custom word cloud.
3 |Prepare the photoThis step is not completed in python. You need to find an image to use. There are a lot of free stock photo sites to pick from like Unsplash, Pixabay, and Pexels to choose from. Once you have found a photo — it needs to be converted to black & white. The black portions of the photo will be where words are displayed, the white areas will show as white. You can complete this in photo editing software such as Photoshop or online with free photo editing software like Pixlr. I’ve used both to pretty easily separate out the background from the subject of the image. Some photos require a little more time than others.
Pro tip: Make sure when creating your photo, that the background is indeed white, and not transparent. The transparent pixel values will be imported as 0, or black, and will not give you a masked image.
3 |Import Photo and Create the WordCloudThere are multiple ways you can color the words in your word cloud. I will go through the details of two, you can see the third in the Github repo. The first will be utilizing a colormap.
Ok, let’s walk through this code. The first line imports your black and white image and the second line adjusts any slight variance in color when creating your image. I had Adobe Illustrator give me a larger range of ‘white’ pixel values between 240 and <255 creating an image that would not work. So this line turns all pixel values greater than 3 white, and the rest are their original values. We then create the WordCloud object with some new parameters.
font_path: this is a path to a font that you would like to use for the words. You can point this path to the font folder of your OS. (ie. C:\Windows\Fonts\font.tff)contour_color: the color of the outline of the imagemax_words: the maximum number of words to use in the imagemask: your image matrixcolormap: the color map to use for the wordscontour_width: the thickness of the outline
It looks great! Though the light colors are pretty hard to read. We could change the background or pick a different color or colormap. Another option is to pick the darker portion of the color map we want to use. We will add two lines that will import a colormap from matplotlib as a matrix of colors, then select the darker part of that matrix.
You can see that the completed product looks pretty good. The image on the right is the image from the code above with the darker colormap. I also used it to create the title image. The image on the left is a custom color. You can find the code for this word cloud in the Github repo. You can save the image using the to_file() method and passing a location to save the file. wordcloud.to_file(“path_to\\wordcloud_image.png”)
3.1 |Create an Image-Colored WordCloudAnother option is to use the colors of the photo itself to color the words. You’ll need to prep the photo as before but only remove the background, and leave the part of the image you would like the words to cover.
We create the mask again (without changing any values). A couple of changes were made to the WordCloud parameters. Some changes to the size and colors of the font and background were made to increase readability. The ImageColorGenerator is used to create the colors for the word cloud, and the recolor() method is used to change the color of the words.
We explored a few different ways to extract some meaningful text utilizing different stopwords from both full text and word frequencies. We also took a look at leveraging log odds ratios to find common words from a portion of the text. We also drew several types of word clouds, using different masks, fonts, and colors. Hopefully, this will help you create some useful visuals for a project.
Check out the Github repository here with the code to all the examples above.
If you are curious about learning and implementing other NLP techniques to extract insights from text, check out this blog post, by Neptune.ai, that covers more than 7 other NLP techniques including sentiment analysis and parts of speech tagging.
If this helped out, let me know! I’d love to hear your thoughts on word clouds and see some of your results.