Job Title Analysis in python and NLTK | by Adonis Gaitatzis | Towards Data Science

A job title indicates a lot about someone’s role and responsibilities. It says if they manage a team, if they control a budget, and their level of specialization.
Knowing this is useful when automating business development or client outreach. For example, a company that sells voice recognition software may want to send messages to:
CTOs and technical directors informing them of the price and benefits of using the voice recognition software.
Potential investors or advisors messages inviting them to see the company’s potential market size.
Founders and engineers instructing them how to use the software.
Training a software to classify job titles is a multi-text text classification problem. For this task, we can use the Python Natural Language Toolkit (NLTK) and Bayesian classification.
First, let’s conceptualize what a job title represents. Each job has a level of responsibility for some department.
Responsibilities include maintenance tasks such as cleaning the floor, management task such as team leadership, or strategy tasks such as deciding on budgets.
Departments include growth-based such as business development and marketing, compliance-based such as finance and legal, or execution-based such as product development and operations.
The responsibility and department are coded into many job titles, like this:
We want to classify each job description into one of these responsibilities and departments.
The first step is to get a list of job titles. There is a list of approximately 800 job titles available on CareerBuilder.
The job titles are paginated so the only way to get them all is to click on each page.
Once the job titles are imported, we can describe the responsibility and department of these jobs.
For this project, we will need the Natural Language Toolkit (NLTK). It contains text processing resources, machine learning tools, and more.
We will be using a Bayesian Multi-Class classifier and the Natural Language Toolkit to classify these job titles.
Import NLTK like this:
import reimport nltkfrom nltk.corpus import stopwordsimport pandas as pdstop_words = set(stopwords.words('english'))
Stop words are common words such as a, and, and the which have little significance to our job titles. NLTK provides a list of English stop words which we will filter out of each job title.
When we get job titles in the wild, for instance from LinkedIn or from Conferences, they sometimes include multiple job titles separated by commas, dashes, colons, or other characters. For simplicity, I’ll show how to extract the first job title:
def get_first_title(title):    # keep "co-founder, co-ceo, etc"    title = re.sub(r"[Cc]o[\-\ ]","", title)    split_titles = re.split(r"\,|\-|\||\&|\:|\/|and", title)    return split_titles[0].strip()
The Bayesian classifier will need a list of features — in this case, which words are in which job description. I also added the first and last words to the features because sometimes the first or last word has significance to the job title. For instance, Director of Marketing and Marketing Director are different roles.
def get_title_features(title):    features = {}    word_tokens = nltk.word_tokenize(title)    filtered_words = [w for w in word_tokens if not w in stop_words]     for word in filtered_words:        features['contains({})'.format(word.lower())] = True    if len(filtered_words) > 0:        first_key = 'first({})'.format(filtered_words[0].lower())        last_key = 'last({})'.format(filtered_words[-1].lower())        features[first_key] = True        features[last_key] = True    return features
This will produce a list of words for a job title, including the first and last words, for example with the job title, Director of Global Operations:
{    "contains(director)": True,    "contains(global)": True,    "contains(operations)": True,    "first(director)": True,    "last(operations)": True}
Now we need to clean and prep the list of job titles, responsibilities, and departments for the classifier.
Presume we have our raw job title data formatted like this:
raw_job_titles = [    {        "title": "Business Development",        "responsibility": "Strategy",        "department": "Business Development"    },    {        "title": "Inside Sales Consultant",        "responsibility": "Execution",        "department": "Sales"    },    ...]
Next, we can train the Bayesian classifier on these datasets. We do this by catalog the features in each job title.
# Responsibilitiesresponsibilities_features = [    (         get_title_features(job_title["title"]),         job_title["responsibility"]    )    for job_title in raw_job_titles    if job_title["responsibility"] is not None]# Departmentsdepartments_features = [    (         get_title_features(job_title["title"]),         job_title["department"]    )    for job_title in raw_job_titles    if job_title["department"] is not None]
We can split the features so that we can train the classifier using some of the features and test the classifier with the rest.
# Responsibilitiesr_size = int(len(responsibilities_features) * 0.5)r_train_set = responsibilities_features[r_size:]r_test_set = responsibilities_features[:r_size]responsibilities_classifier = nltk.NaiveBayesClassifier.train(    r_train_set)print("Responsibility classification accuracy: {}".format(    nltk.classify.accuracy(        responsibilities_classifier,        r_test_set    )))# Departmentsd_size = int(len(departments_features) * 0.5)d_train_set = departments_features[d_size:]d_test_set = departments_features[:d_size]departments_classifier = nltk.NaiveBayesClassifier.train(    d_train_set)print("Department classification accuracy: {}".format(    nltk.classify.accuracy(        departments_classifier,        d_test_set    )))
The output should resemble this:
Responsibility classification accuracy: 0.80588235294117646Department classification accuracy: 0.80526315789473684
If you are happy with the results, you can classify new job titles.
title = "Director of Communications"responsibility = responsibilities_classifier.classify(    get_title_features(title))department = departments_classifier.classify(    get_title_features(title))print("Job title: '{}'".format(title))print("Responsibility: '{}'".format(responsibility))print("Department: '{}'".format(department))
Which results in the following output:
Job title: 'Director of Communications'Responsibility: 'Strategist'Department: 'Marketing'
One limitation is that the classifier will assign a department and responsibility to every job title, regardless of how ambiguous the job title is. This leads to the classification of job titles that should remain unclassified or reviewed by a person.
To overcome this, we can reveal the classifier’s confidence in its interpretation so that we can choose whether or not to accept it.
# Responsibilityresponsibility_probability = \    responsibilities_classifier.prob_classify(        get_title_features(title)    )responsibility_probability = 100 * responsibility_probability.prob(    responsibility_probability.max())print("Responsibility confidence: {}%".format(    round(responsibility_probability)))# Departmentdepartment_probability = \    departments_classifier.prob_classify(        get_title_features(title)    )department_probability = 100 * department_probability.prob(    department_probability.max())print("Department confidence: {}%".format(    round(department_probability)))
The result is something like this:
Responsibility confidence: 86%Department confidence: 79%
Source code can be found on Github. I look forward to hearing any feedback or questions.