Clustering documents with Python. A simple example with Wikipedia... | by Dimitris Panagopoulos | Towards Data Science

Natural Language Processing has made huge advancements in the last years. Currently, various implementations of neural networks are cutting edge and it seems that everybody talks about them. But, sometimes a simpler solution might be preferable. After all, one should try to walk before running. In this short article, I am going to demonstrate a simple method for clustering documents with Python. All code is available at GitHub (please note that it might be better to view the code in nbviewer).
We are going to cluster Wikipedia articles using k-means algorithm. The steps for doing that are the following:
fetch some Wikipedia articles,
fetch some Wikipedia articles,
2. represent each article as a vector,
3. perform k-means clustering,
4. evaluate the result.
Using the wikipedia package it is very easy to download content from Wikipedia. For this example, we will use the content of the articles for:
Data Science
Artificial intelligence
Machine Learning
European Central Bank
Bank
Financial technology
International Monetary Fund
Basketball
Swimming
Tennis
The content of each Wikipedia article is stored wiki_list while the title of each article is stored in variable title.
import pandas as pdimport wikipediaarticles=['Data Science','Artificial intelligence','Machine Learning','European Central Bank','Bank','Financial technology','International Monetary Fund','Basketball','Swimming','Tennis']wiki_lst=[]title=[]for article in articles:   print("loading content: ",article)   wiki_lst.append(wikipedia.page(article).content)   title.append(article)print("examine content")wiki_lstAt the top of the github page there is a button that will allow you to execute the notebook in Google’s Colab. Feel free to experiment!
Since we are going to use k-means, we need to represent each article as a numeric vector. A popular method is to use term-frequency/inverse-document-frequency (tf-idf). Put it simply, with this method for each word w and document d we calculate:
tf(w,d): the ratio of the number of appearances of w in d divided with the total number of words in d.
idf(w): the logarithm of the fraction of the total number of documents divided by the number of documents that contain w.
tfidf(w,d)=tf(w,d) x idf(w)
It is recommended that common, stop words are excluded. All the calculations are easily done with sklearn’s TfidfVectorizer.
from sklearn.feature_extraction.text import TfidfVectorizervectorizer = TfidfVectorizer(stop_words={'english'})X = vectorizer.fit_transform(wiki_lst)
(To be honest, the calculation of sklearn is slightly different. You can read about in the documentation.)
Each row of variable X is a vector representation of a Wikipedia article. Hence, we can use X as input for the k-means algorithm.
First, we must decide on the number of clusters. Here, we will use the elbow method.
import matplotlib.pyplot as pltfrom sklearn.cluster import KMeansSum_of_squared_distances = []K = range(2,10)for k in K:   km = KMeans(n_clusters=k, max_iter=200, n_init=10)   km = km.fit(X)   Sum_of_squared_distances.append(km.inertia_)plt.plot(K, Sum_of_squared_distances, 'bx-')plt.xlabel('k')plt.ylabel('Sum_of_squared_distances')plt.title('Elbow Method For Optimal k')plt.show()
The plot is almost a straight line, probably because we have to few articles. But at a closer examination a dent appears for k=4 or k=6. We will try to cluster into 6 groups.
true_k = 6model = KMeans(n_clusters=true_k, init='k-means++', max_iter=200, n_init=10)model.fit(X)labels=model.labels_wiki_cl=pd.DataFrame(list(zip(title,labels)),columns=['title','cluster'])print(wiki_cl.sort_values(by=['cluster']))
Since we have used only 10 articles, it is fairly easy to evaluate the clustering just by examining what articles are contained in each cluster. That would be difficult for a large corpus. A nice way is to create a word cloud from the articles of each cluster.
from wordcloud import WordCloudresult={'cluster':labels,'wiki':wiki_lst}result=pd.DataFrame(result)for k in range(0,true_k):   s=result[result.cluster==k]   text=s['wiki'].str.cat(sep=' ')   text=text.lower()   text=' '.join([word for word in text.split()])   wordcloud = WordCloud(max_font_size=50, max_words=100, background_color="white").generate(text)   print('Cluster: {}'.format(k))   print('Titles')   titles=wiki_cl[wiki_cl.cluster==k]['title']            print(titles.to_string(index=False))   plt.figure()   plt.imshow(wordcloud, interpolation="bilinear")   plt.axis("off")   plt.show()
Cluster 0 consists of articles on European Central Bank, Bank and International Monetary Fund
Cluster 1 consists of articles on Artificial intelligence and Machine Learning
Cluster 2 has an article for Swimming
Cluster 3 has an article for Data Science
Cluster 4 has articles on Basketball and Tennis
Cluster 5 has an article on Financial Technology
It might seem odd that swimming is not in the same cluster as basketball and tennis. Or that AI and Machine Learning are not in the same group with Data Science. That is because we choose to create 6 clusters. But by looking at the word clouds we can see that articles on basket and tennis have words like game, player, team, ball in common while the article on swimming does not.
At the top of the GitHub page, there is a button that will allow you to execute the notebook in Google’s Colab. You can easily try a different number of clusters. The results might surprise you!