Try TextHero: The Absolute Simplest way to Clean and Analyze Text in Pandas | by Eric Kleppen | Towards Data Science

I am always on the lookout for new tools to help me simplify my natural language processing pipelines, so when I stumbled upon a short video clip showing Texthero’s functionality I knew I had to try it right away. Texthero is designed as a Pandas wrapper, so it makes it easier than ever to preprocess and analyze text based Pandas Series. Immediately, I pulled up the docs, opened a notebook and downloaded a couple thousand Reddit threads to analyze to give the new lib a test.
NOTE: The Texthero library is still in beta! There might be bugs, and pipelines might change. I found a bug in the wordcloud functionality and reported it. It should be fixed in a coming update!
I will look at these a lot closer when working with the code, but for a quick overview, Texthero is broke into four modules of functionality:
The Preprocessing module is all about efficiently cleaning text-based Pandas series. It is primarily using Regular Expressions (regex) under the hood.
The NLP module houses a few common NLP tasks, like Named Entity Recognition and Noun Chunks. It is using Spacy under the hood.
The Representation module is used to create word vectors using different algorithms. It includes things like Principle Component Analysis and kMeans as well. It uses scikit-learn for TF-IDF and Count, and embeddings are loaded pre-computed from language models.
The Visualization module is used to visualize the representations in a scatter plot or generate wordclouds. This module currently has only a few features and uses Plotly and WordCloud under the hood.
Review the documentation for a complete list of features!
Text preprocessing, representation and visualization from zero to hero.texthero.org
Although installation was easy using pip, I encountered a conflict when trying to install it in my environment that has Apache Airflow due to a pandas version problem. Also, it took a while to install in a new environment since it uses so many other libraries on the backend. It downloads a couple additional things after importing for the first time too.
For the dataset, I am using PRAW to pull data from Reddit. Check this article if you need a PRAW refresher.
towardsdatascience.com
!pip install textheroimport prawimport pandas as pd import texthero as herofrom config import cid, csec, ua #PRAW credentials
Notice when you import Texthero for the first time, you will see it downloads a few things from NLTK and Spacy:
I am pulling data from the Teaching subreddit to see if we can identify any themes around concern for starting school up in the fall in COVID heavy America.
#create a reddit connectionreddit = praw.Reddit(client_id= cid,                     client_secret= csec,                     user_agent= ua)#list for df conversionposts = []#return 1000 new posts from teachingnew = reddit.subreddit('teaching').new(limit=1000)#return the important attributesfor post in new:    posts.append([post.title, post.score, post.num_comments, post.selftext, post.created, post.pinned, post.total_awards_received])#create a dataframedf = pd.DataFrame(posts,columns=['title', 'score', 'comments', 'post', 'created', 'pinned', 'total awards'])#return top 3 df rowsdf.head(3)
PRAW makes it super easy to pull data from Reddit and load it into a pandas dataframe. Notice I pull 1000 new posts from teaching. The dataframe output will look something like this using df.head(3):
The real selling point of the texthero library is its simplified preprocessing pipeline. Can’t remember regex syntax? Texthero has you covered! Simply call the .clean() method and pass the dataframe series:
df['clean_title'] = hero.clean(df['title'])
It runs the following seven functions by default when using clean()
fillna(s) Replace not assigned values with empty spaces.lowercase(s) Lowercase all text.remove_digits() Remove all blocks of digits.remove_punctuation() Remove all string.punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).remove_diacritics() Remove all accents from strings.remove_stopwords() Remove all stop words.remove_whitespace() Remove all white space between words.
fillna(s) Replace not assigned values with empty spaces.
lowercase(s) Lowercase all text.
remove_digits() Remove all blocks of digits.
remove_punctuation() Remove all string.punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).
remove_diacritics() Remove all accents from strings.
remove_stopwords() Remove all stop words.
remove_whitespace() Remove all white space between words.
If the default doesn’t do what is needed, creating a custom cleaning pipeline is super simple. For example, if I want to keep stop-words and stem the included words, I can comment out remove_stopwords and add texthero.preprocessing.stem() to the pipeline:
from texthero import preprocessing#create a custom cleaning pipelinecustom_pipeline = [preprocessing.fillna                   , preprocessing.lowercase                   , preprocessing.remove_digits                   , preprocessing.remove_punctuation                   , preprocessing.remove_diacritics                   #, preprocessing.remove_stopwords                   , preprocessing.remove_whitespace                   , preprocessing.stem]#pass the custom_pipeline to the pipeline argumentdf['clean_title'] = hero.clean(df['title'], pipeline = custom_pipeline)df.head()
Notice the custom_pipeline is a list of preprocessing functions. Check out the documentation for the full list of preprocessing features!
Inspecting top words is only 1 line of code, and that is something I like to do to see if there are additional words I should consider adding to the stop words list. Texthero doesn’t yet have bar graphs built into it, it only has scatter, so I’ll use Plotly express to visualize the top words in a bar graph.
tw = hero.visualization.top_words(df['clean_title']).head(10)import plotly.express as pxfig = px.bar(tw)fig.show()tw.head()
Stemming the words and adding “ ‘ “ (notice it between teachers and students) to the stop words should give me more unique words. Stemming was already added to the custom pipeline, but the stop words need to be added. Stop words can be added to the stop word list using union on the two lists:
from texthero import stopwordsdefault_stopwords = stopwords.DEFAULT#add a list of stopwords to the stopwordscustom_stopwords = default_stopwords.union(set(["'"]))#Call remove_stopwords and pass the custom_stopwords listdf['clean_title'] = hero.remove_stopwords(df['clean_title'], custom_stopwords)
Notice the custom_stopwords list is passed into hero.remove_stopwords(). I’ll re-visualize it and check the results!
The results look a bit better after stemming and additional stop words have been applied!
Thanks to Panda’s .pipe(), chaining the Texthero module components together is super simple. To visualize the title, I am going to use Principle Component Analysis to compress the vector space. I am also going to run K-means clustering to add color. Remember, Texthero takes a Series as input and Series as output so I can set the output to be a new column in the dataframe.
#Add pca value to dataframe to use as visualization coordinatesdf['pca'] = (            df['clean_title']            .pipe(hero.tfidf)            .pipe(hero.pca)   )#Add k-means cluster to dataframe df['kmeans'] = (            df['clean_title']            .pipe(hero.tfidf)            .pipe(hero.kmeans)   )df.head()
PCA and K-means clustering have been applied using only a few lines of code! Now the data can be visualized using hero.scatterplot()
#generate scatter plothero.scatterplot(df, 'pca', color = 'kmeans', hover_data=['title'] )
Since it uses Plotly under the hood, the scatter plot is as interactive as you’d expect! It can be zoomed in as needed. Now it is time to explore visualised results and see what insights can be gained!
Although the library is still in beta, I see a promising future for Texthero and hope it gets the love it deserves. It makes cleaning and preparing text in Pandas dataframes a breeze. I hope a few more visualization options get added, but the scatter plot is a great start. If you’re interested in learning more about Natural Language Processing, check out my other articles covering some basic and advanced topics.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Thanks for reading. Here is the complete code:
#create a reddit connectionreddit = praw.Reddit(client_id= cid,                     client_secret= csec,                     user_agent= ua)#list for df conversionposts = []#return 1000 new posts from teachingnew = reddit.subreddit('teaching').new(limit=1000)#return the important attributesfor post in new:    posts.append([post.title, post.score, post.num_comments, post.selftext, post.created, post.pinned, post.total_awards_received])#create a dataframedf = pd.DataFrame(posts,columns=['title', 'score', 'comments', 'post', 'created', 'pinned', 'total awards'])#return top 3 df rowsdf.head(3)from texthero import preprocessingcustom_pipeline = [preprocessing.fillna                   , preprocessing.lowercase                   , preprocessing.remove_digits                   , preprocessing.remove_punctuation                   , preprocessing.remove_diacritics                   , preprocessing.remove_stopwords                   , preprocessing.remove_whitespace                   , preprocessing.stem]df['clean_title'] = hero.clean(df['title'], pipeline = custom_pipeline)df.head()from texthero import stopwordsdefault_stopwords = stopwords.DEFAULTcustom_stopwords = default_stopwords.union(set(["'"]))df['clean_title'] = hero.remove_stopwords(df['clean_title'], custom_stopwords)hero.visualization.top_words(df['clean_title'])tw = hero.visualization.top_words(df['clean_title']).head(10)import plotly.express as pxfig = px.bar(tw)fig.show()df['pca'] = (            df['clean_title']            .pipe(hero.tfidf)            .pipe(hero.pca)   )df['kmeans'] = (            df['clean_title']            .pipe(hero.tfidf)            .pipe(hero.kmeans)   )hero.scatterplot(df, 'pca', color = 'kmeans', hover_data=['title'] )
If you enjoyed this, follow me on Medium for more
Get FULL ACCESS and help support my content by subscribing
Let’s connect on LinkedIn
Analyze Data using Python? Check out my website
— Eric Kleppen