Three Methods for Comparing Text Data to Instantly Boost the Impact of Your Analysis | by Andrew Hong | Towards Data Science

(The data and full Jupyter notebook walk-through can be found here.)
If you’re looking for a job as a data analyst or scientist, as well as trying to learn NLP — get ready to knock out two birds with one stone! For this article, I’ll be using 1000 job postings for “Data Analyst” on Indeed.com as my set of documents (corpora). I chose data analyst since there will be a larger variety of postings and requirements compared to data scientist. The columns are pretty self-explanatory:
Before we dive into code, let’s stop and think about why we’re analyzing text data, to begin with. For me, it’s usually for one of two reasons:
I want to find the area of greatest overlap in my corpora, i.e what are the relationships and variations between documentsI’m looking for a set of keywords or values in a subset of my corpora, i.e. what is the main subject or points of discussion across documents
I want to find the area of greatest overlap in my corpora, i.e what are the relationships and variations between documents
I’m looking for a set of keywords or values in a subset of my corpora, i.e. what is the main subject or points of discussion across documents
In both cases, the point is to help me narrow my scope of research and save me the time of reading through everything myself. Of course, these methods also help as a precursor to building a machine learning pipeline, as the analysis may lead you to errors or biases in your text data before you start training your model.
Euclidean Distance of Tokenized Vectors with PCA and t-SNENormal and Soft Cosine Similarities of a Set of CorporaExtracting Structured Relationships from Unstructured Text Data
Euclidean Distance of Tokenized Vectors with PCA and t-SNE
Normal and Soft Cosine Similarities of a Set of Corpora
Extracting Structured Relationships from Unstructured Text Data
Let’s start with the basics, tokenizing the data using nltk methods:
And then apply CountVectorizer (think of this as one-hot encoding) as well as Tfidf transformer (think of this as weighting the encoding, so more frequent words get lower weight).
Ultimately, we can fit it into a PCA model and plot the first two components using Seaborn’s KDE plot:
This is great for getting an initial idea of lay of the land, based on usage of words alone. However, since we aren’t layering any categories such as industry or salary on top of this, we won’t get too much more insight from this step.
For PCA based on euclidean distance covering large corpora, you usually will not end up finding strong relationships or clusters. This is because semantic meaning of words has not yet been considered, on top of the fact some posting are two lines long and others are multiple paragraphs. We can solve both of these issues with Cosine Similarity, or the measurement of angles between word vectors.
There are two main options for cosine similarity. The first choice is to take the cosine similarity on the word vectors after applying all the same tokenization steps as we did before and create a matrix. Simple enough:
The second or going a step further and taking an inner dot product with L2 normalization between a previous word embedding model similarity matrix and the normal similarity matrix we just calculated to get a soft cosine matrix (essentially trying to capture semantic meaning of words in the descriptions better). We can do this easily using gensim word2vec or FastText pre-trained word embedding models. If you are unfamiliar with embedding models, think of it as words that given a similarity score based on how similar the context of use is (i.e. the two words directly before and after it).
We get an average similarity is 0.80 if we take cosine.mean().mean() , which makes sense given these are all data analyst postings. Taking the same PCA and density plot gives us:
With the added standardization and semantic layering, we can now dig into certain most dense areas to see what posts are clustering there.
Using the above search function on an area of the plot x(-1,1) and y(-0.5,-0.1), then applying gensim summarize gives us:
Manage and consolidate all Operational Spreadsheets.Work with management to prioritize business and information needs.Locate and define new process improvement opportunities.Collaborating with the executive team to understand the analytical needs of our multichannel operations and developing data-driven insights that are both strategic and operational.Formulating and championing insights on specific business tactics such as inventory forecasting and gaps analysis to drive those insights into action.What You Need to Succeed:-  to  years of experience working in Inventory Analyst role or Warehouse Master Data Management.- Bachelor's degree or higher education in Computer Science, Information Technology, Statistics, Business Administration or a closely related field.- Sound knowledge of inventory control practices & supply chain- Advanced Excel spreadsheet knowledge: Pivot Table, VLOOKUP, Various Formula- Familiarity with SAP and Warehouse Management System is a plus.Other Skills:-Exceptional written and oral communication skills.- Eye for detail and have a strong analytical skills.- Ability to priorities work effectively.- Able to balance/challenge conflicting objectives.- Able to see the bigger picture beyond the scope of their role.- Able to challenge the current status quo and being open minded towards new and different ways of doing things.- Tackles new problems in a timely manner. Pharmapacks, is a leading e-commerce company with a proprietary tech platform that empowers Brands with a complete and cost-effective logistics, fulfillment, marketing and sales solution.
While a summary of 81 postings may not read entirely coherently, you can see that this area is focused on management of data and excel spreadsheets in a logistics/inventory environment. Since I already stored the URL’s, I could go on and apply to just these 81 postings if I was interested!
A lot of useful data is hidden inside reports and articles, instead of tables. For example, job postings often list requirements and salaries on top of a brief description of the role.
I’ll be focusing this analysis on the skills and tasks a job posting is asking for. We can do this by deconstructing sentences into parts of speech, then labeling the main subject and object as well as the relationship between them.
For example,
“This position will be the first of its kind at (Company Name) and as such, the person in this role must be able and excited to take full responsibility for building out and maintaining internal tracking and reporting procedures”
can be broken down into parts and dependencies using spaCy’s english model:
By leveraging functions that create entity pairs as well as relationships (based on this helpful article) we can get a better picture of what types of tasks are being asked of data analysts. Taking value_counts() of the relationships found gives us:
Filtering for “Provide” and creating a network graph gives us:
The knowledge graph works well for “Provide” as there are only 100 mentions, but can vary for larger counts of relationships. We can see that analysts are expected to provide everything from potential business risks to action plans and ad hoc reports.
I hope this has helped you understand how to better approach text data. With these three tools, we’ve quickly identified niche types of roles within “data analyst”, as well as some of the abilities and tasks you should prepare for them. If you want to see deeper analysis using these techniques, check out this article!
As mentioned earlier, the data and full Jupyter notebook walk-through can be found here.