Data Science job search: Using NLP and LDA in Python
By: Thomas Caffrey
Tags: NLP, Python, Topic Modeling, Data Science, Machine Learning

2. Analysing the data

Volume of posts from Companies / Recruiters

Unsurprisingly most of the frequently positing companies are recruiters. Although in greater London it seems Harnham and Datatech Analytics have the largest number of roles. It may be worth speaking directly with these 2 to understand more about the roles they contain.

Removing duplicate postings

Sometimes recruiters or companies post the same advert for a job which results in duplicate data. These can simply be removed based on the job description however if competing recruiters are posting adverts for the same job there can be slight differences.

The solution that seemed the easiest to remove these closely matched jobs was to compare cosine similarities based on their word counts. Any that contained a score close to 1 without equalling 1 would almost certainly be a repetition of the same job.

Removing similar job descriptions with cosine similarity

Cosine similarity is a metric used to determine how similar the documents are irrespective of their size. It measures the cosine of the angle between two vectors (containing word counts) projected in a multi-dimensional space, where each dimension corresponds to a word in the document. The cosine similarity captures the orientation (the angle) of the documents and not the magnitude. The cosine similarity is advantageous because if two similar documents are far apart by the Euclidean distance due to their respective size they could still have a small angle between them, which indicates they are similar.

These job adverts are transformed to a vector based on their word counts before calculating their cosine similarities. This method will definitely catch any documents almost exactly the same but should also catch any that have had an extra section added and changed the length of the post (another recruiter posting the same job description in addition to information related to the recruitment company for example).

I made a few checks by inserting a reasonably long unrelated recruiter contact information into adverts that had a high cosine similarity with another (close to 1). With this “noise” the cosine similarity reduced but still returned a cosine similarity >0.99 which suggests these similar job description should still be picked up. However the Euclidean distance had a noticeable increase which indicates why it might not be worth using a distance measure to remove similar documents.

Some exploration on closely matched scores resulted in the cutoff of above 0.98 cosine similarity being used as a removal. This cutoff level captures adverts that have slight variations but are essentially the same job posting.

Job Titles

Data Scientist or machine learning engineer are the highest returning results for job title. However of the approximate 600 results returned from the search it seems there’s a lot of variation in titles used and approximately 400 job titles only used once.

For now I will class each job titles into the following categories:

Lead: Any title containing lead, chief, head or manager

Senior: Any title containing senior or principal

Graduate: Any title containing graduate

Regular: Will class anything else as a regular role

I’m sure there will be some roles that end up getting classed as regular when in reality they are a more senior or junior role. However this seems currently to be a better way to narrow down jobs based on titles. I at least know for certain if a role is Senior, Lead or Graduate that it is not required in the set of results I need for narrowing down my job search.

Contract Type

A mixture of contract and permanent roles are contained on Indeed. 40% of roles have a type of contract associated with them. It may be possible to further investigate the ones without a contract type and find out more from their description. For now I will class contract types as the following:

Contract: Anything that has ‘contract’ or ‘temporary’ associated with it

Apprenticeship

Internship

Part-Time

Permanent: Any contract that is classed as ‘Full-time’ or ‘Permanent’

Python vs R

Looking into whether Python or R is specified in the job description is more out of curiosity.

Python is overwhelmingly the program of choice in the job descriptions. This is higher than I thought it would be and makes the case for using Python over R for producing data science related analysis / models.

Salary

On Indeed permanent roles have a salary per year associated with them but for contractor roles the rate (hourly, daily,weekly, monthly) is specified.