Use BeautifulSoup and feature extraction in Python to automate the job search process
Tags: Job Search, Web Scraping, Beautifulsoup, Count Vectorizer, Data Science

Often when a Data Engineer is pitched with an automation problem for a manhours intensive clerical, repetitive data processing task, it is framed from the perspective of the business owner. While there are applications availabe that screen resumes for the hiring manager, what options are there to screen job posts?

Many of the tips for job search published online by career professionals (Muse, Indeed) focus on either optimizing the resume, or the interview process. These are the process steps where the candidate is interacting with the hiring manager. What gets less attention is the first part of the process — evaluate the market opportunities and honing a shortlist of potential matches.

Reducing the share of the candidate’s attention that is spent on the task of screening through jobs unlocks more bandwidth for the higher value added process that these career professionals are promoting— researching the positions, company, engaging with people in the company, the hiring manager and upgrading your qualifications. Some platforms already offer this heavy lifting step for the candidates. LinkedIn provides suggested matches for both hiring managers AND candidates. Not every platform offers this feature however. Specifically, Singapore job portal MyCareerFutures does not appear to have a search-by-match feature.

The job search process

The first step of any automation task is to get a handle on the problem you are trying to solve — its definition, constraints, and associated process workflow(s).

Problem statement

Minimize the non-value added time searching for a job match, subject to the constraints of the candidate qualifications and the available openings

Process workflow

One way of describing the job search steps are :

(A) search for jobs

(B) apply for job

(C) get invited for an interview

(D) attend rounds of interviews

(E) job offer!

Once a shortlist of jobs is ready for application (B) and the process moves to engaging agent and/or hiring manager ( C), there is less scope for pure data processing to improve the quality of the outcomes. Before reaching step C, while there may be some factors outside of the candidate’s control related to the market or structural biases in the hiring process, the part that is within the candidates control affecting likelihood of positive response are :

Step A : Qualification match % between the job post and the candidate CV

Step B : Quality of the application submitted (no typos, etc..)

The objective of steps A and B is to maximize the likelihood of a positive response from the hiring manager in step C. Between the search (A) and applying for jobs (B) the scope most appropriate for a data processing application to be value-adding is Step A, the job search, which can be broken into 3 sub-steps.

Fetch : job openings from a listing source Screen : the jobs using either the built-in software from the listing provider or offline Shortlist : Manually review the shortlisted jobs to decide which ones to apply for

The overall process can be framed as a learning problem with real feedback from the actual response rate as a basis for tuning the upstream process steps of screening, shortlisting and applying for the jobs.

Performance specification

To design a program that accomplishes something, the problem needs to be translated into useful performance metrics that unambiguously define whether the program has succeeded or failed. One meaningful metric to monitor that bridges the two phases of the search is

callback volume = # positive responses per week

The main goal of the program is to help the candidate economize their limited attention and total duration of the search process. Every step from the automated screening onwards takes up the candidate’s attention and thus an opportunity cost for other value adding activities. The response rate of hiring managers has its own speed that is outside of the candidate’s influence, typically 2–6 weeks. Give a desired 90% probability of securing an offer in 4 months and a hypothetical 15% job offer per callback conditional success rate that sets a minimum requirement of :

~ 1 callback per week

For a given callback volume and raw openings (fetch volume), there is an inverse trade-off between the response rate (callbacks per job applied) and the capture rate of shortlisted jobs from the raw fetch from source.

For a raw fetch rate of 2,000 posts/week and a positive response rate of ~10% (typical values reported in the UK are 10–25% depending on national origin), to achieve 1 callback per week, the minimum overall capture rate should be

capture rate ≥ 1%

Two steps : Fast screen → Slow review

2,000 posts/week is a large volume to process. Attention bandwith is a real constraint, especially for one person. The natural response for most candidate faced with this general problem would be to generate a big list and quickly scan the list in rapidfire and then run through the shortlist in a more deliberate second-pass review.

The first scan step plays to human perception strengths. This step is also part of the function that professional networks perform to leverage the candidate’s efforts. The scan step is also where a data processing algorithm has the most potential for value improvement as it can be almost completely substituted. The eyeball scan on a few samples could set as the benchmark performance for the automated version on speed and qualification match% quality.

The scan technique alone without some kind of other supporting process is a crude substitute though for a profile assessment of qualification match. Once it comes to applying for a job, one way or another the candidate is going to transition to a slower, more deliberate review of the post. So while the second review step can be streamlined, unlike the scan step it is unlikely to be completely substituted by automation.

So a useful way of thinking of the search is a two-step process of a fast imprecise screen performed by software, followed by a slow deliberate shortlist review performed by a human. The overall capture rate is the product of the capture rates for these two steps :

overall capture = screen capture % * shortlist capture %

There is a trade-off between the two capture rates. Being specific about what you are looking for is an asset. For even moderate shortlist assessment time of 6 min per post, this can quickly explode into a manhour bottleneck for one person. While 20 posts, 2 hrs a week and even 60 posts, 6 hrs a week are manageable, 400 posts, 40 hrs a week is infeasible for even the most ambitious candidates. A liberal screening algorithm that is timid to throw out marginal matches in order to allow for more control in the deliberate shortlist step would ultimately be counter-productive. This would only shift the burden onto the resource intensive deliberate shortlist review and end up bottlenecking the volume capacity of the overall process.

For a raw fetch rate of 2,000 posts a week and a limit of < 60 posts a week for shortlisting translates to a screening selectivity of < 3% .

Program specification :

maximize the match % given a selectivity of 1–3%.

So that becomes the specification for the automated screening program — to maximize the match % likelihood given a selectivity of 1–3%. The rest is up the the candidate.

Search criteria

A useful screening algorithm should inform the candidate what is available in the job market and incorporate the candidate’s qualifications. The framework of qualification features for a job profile are not pre-determined and up to the design of the program.

A simple framework to start with proposed in this post uses 4 feature — Rank, Role, Industry and Specialization. For this example the candidate is a mid-level career Data Analyst / Consultant with some python experience and scientific/engineering background.

Six search criteria (industries) that might approximate such a profile

industries = [‘Consulting’,’Engineering’,’Professional Services’, ‘Science’,’Environment’,’Information Technology’,’Manufacturing’]

For a candidate first starting out, it might be helpful to get a broad overview of the available space in terms of # available posts and salary range. The search conditions of minimum salary of S$6,000 per month and the 6 industry keywords returns ~ 2,000 posts/week on the MCF website.

Above the S$6k/month salary limit, broadly the overall salary distribution for all industries is roughly linear from 6,000 until 11,000 where it starts to tail-off into the more exclusive senior management positions.

Within this broad search criteria, the largest by volume industry search terms for the Singapore job market in Nov 2020 are Engineering (27%) Environment (26%), Marketing (11%) and Science (12%). Consulting, Manufacturing, Finance and IT each make-up ~ 5%.

The salary range varies by industry. The (50th-75th pct) aggregate range is (S$3,800–S$6,500) with the top range

Professional Services (S$8,000–S$11,000)

Consulting (S$6,800–S$10,000)

Next top ranges are

IT (S$6,200–S$8,500)

Science (S$6,300–S$8,900)

Engineering (S$4,800–S$7,500)

Finance (Banking) (S$5,000–S$8,800)

Environment (S$4,200–S$7,800) and Marketing (S$4,200–S$7,100) have a wide range and also a large share of the openings (26%, 11% respectively)

Manufacturing (S$4,000–S$6,500) is the lowest search term with 5% of the openings

Rank

The usefulness of salary information is not only for the candidate’s own financial interests, it’s also a good proxy for ‘Rank’, a prominent features of a job profile. The salary range can vary between industries, so to correct for the between-industry variations the within-industry salary percentile provides a more constistient measure across industries. Within the 6 shortlisted industries this rank measure has > 0.9 R2 correlation with years of experience on the range 2–7 years and 0.2–0.9 percentiles. Beyond the 90th percentile the relationship breaks down as further salary increases become decoupled from any theoretical prediction based on qualifications.

Another consideration for the candidate is the age of the post. The standard setting on the website is for posts to remain on the site for 1 month, and this roughly corresponds to how long it takes for the post to saturate with applications. Popular posts can see > 50 applicants in the first week, but on average 70% of openings >1 week old have received at least one applicants and an average of 3.8 applicants, and that figure goes to >90% and >22 applicants by week 3.

So practically a candidate has ~2–3 weeks to respond once the job is posted and it is probably not worthwhile screening posts that are older than 3 weeks since likely by this time the hiring manager has already received enough candidates to shortlist and already extended invitations for interviews.

Automation with Python : jobsearch

A Python module jobsearch was developed to implement the job post screening process. The basic application form includes three basic components : a place to store the job posts (Data), a user interface UX/UI and the python program code. The presentation here is a summary of the design highlights and code snippits. The full application documentation and source code is available on the jobsearch Github wiki.

The application utilizes a local sqlite database on the host PC to store the data and an open-access google spreadsheet (gsheet) for the user interface. Some routines can also be initiated using .bat files on the user’s Desktop. If these .bat files are then added to the Windows Task Scheduler this would give the program more of an ‘automation’ feel although if the candidate is sticking to a regular schedule of shortlisting the posts they could just run the routines on-demand, so this extra automation step is probably not critical.

Screening the job posts

The two automated processes are to fetch, and screen new job posts.

Fetch : new job profiles from the web and store them to memory Screen : the job profiles and push the results to the user via gsheet

The screening process is composed of three main steps — cleanup, text analysis and match score calculation.

Cleanup : and derank the position title Text analysis : on the ‘deranked title’ to produce a ‘title score’ Score : the profile from combination of ‘rank’ and ‘title score’

Once the match score is calculated for each profile, a sub-query of the job profiles gets posted to the user with filter of no older than 3 weeks and match score ≥ threshold. The program can be tuned to meet the selectivity performance specification by adjusting the threshold match score.

Rank and match score

Given that salary and rank are closely associated, for a given industry and role, the rank is the equivalent of the ‘price’ in other markets. Some candidates may also be motivated for an increase in rank within the same industry for reasons beyond salary. For any heiarchially organized industry, the law of supply and demand dictate that there will be more competition for the more scarce set of higher rank positions. Each candidate has a trade-off choice to make between the position rank and the time taken for the search. The candidate can adopt two ways of framing the search, one where rank is the optimization variable to maximize, and the other where the rank, or salary is set as a constraint and the goal is to minimize the search time.

In this example the candidate’s financial preferences have already been accounted for by setting the minimum salary in the query to fetch the raw job posts. Spending time searching and applying for jobs above this salary level threshold may have the reward of a higher pay, but may be more selective and lower the probability of the hiring manager extending an offer. The risk of dragging the search process on for too long may go beyond the direct opportunity cost of lost hrs. The demoralizing effect of too low of a callback response rate may sap the resolve of the candidate from staying motivated to continue the search.

If the match is defined as likelihood of the hiring manager extending a job offer, for a given qualification level, a job with a lower rank might actually improve the likelihood of a ‘match’.

A hypothetical match score that would implements this logic :

Match = 1/rank quantile (integer range=1 to 4) * qualification match %

The algorithm uses the title score to approximate the qualification match % and defined the title score as the count of the # of matching keyword tags found in the title and ranges as an integer from 0–2.

Job posts on MyCareerFutures

MyCareerFutures (MCF) is an online job portal in Singpaore that has a number of useful features for candidates — search by salary, date posted, keyword, etc.. Three features in particular unique to the site and relevant for the screening program are :

Exposed url pattern for search queries High % rate of posts with many of the standard fields complete including salary range which many job portals do not reveal Personalized profile and site generated match % rating which requires Singpass (secure national ID) 2FA login.

screenshot of job search results from MyCareerFutures

jobs report query

For a given set of query conditions {} there is a corresponding unique url. That’s the features of the search query url pattern.

For example, the query url

https://www.mycareersfuture.gov.sg/search?search=Consulting&salary=6000&sortBy=new_posting_date&page=0

corresponds to those query criteria :

{keyword = Consulting, min salary = 6,000}

The general url pattern is continuous string of key, value pairs joined by ‘&’. The sortBy call and page selector are additional conditions to specify the query for how the job results should display.

root/search?criteria1=value1&criteria2=value2&sortBy=new_posting_date&page=n

The page is organized into ‘job cards’ with the default of 20 cards per page (and lots of ads inbetween). Each job card has standard information such as the posted date, short title, company name, and the expected salary range.

Another useful field is the url to the job profile page

job profile

The url for this Java Developer example is

https://www.mycareersfuture.gov.sg/job/information-technology/java-developer-avensys-consulting-2b491691a85457ab6f6e30a851f8d8dc

Each job on MCF has both a unique MCF REF id and a unique urlid. It is straightforward to retrieve a job if the urlid is known.

root/job/urlid

There are ~ 11 information fields on each job profile page,

some of them are also available on the job card. Generally they fall into structured fields and unstructured (human-readable paragraph or sentence) fields.

Structured fields

Rank Years of experience Industry classification # of applicants Closing date (item 10) MySkillsFuture skills match % (item 11) Keywords

Unstructured fields

(item 6) General description (item 7) Formal qualifications (item 8) Transferrable (generic) qualities (item 9) Specilized experience

Together these fields on the job card and the profile are all of the information that a screening program has to work with.

Automated web browsing with Selenium webdriver

There are different forms of publishing information on the internet depending on the purpose. Two common forms are web pages and APIs. A web page presents information in an appealing, interactive form that is optimized for a human interface. APIs are a protocol for how applications talk to one another on the web. Although more web programs are developing both web and API interfaces this represents only a small fraction of the overall space of web pages. Furthermore at the time of writing this post there is no known API interface for MyCareerFutures. The general process of translating encoded information from one langauge to another is known as parsing and parsing from a human readable html source code into structured fields that can be stored into a database for analysis is known as web scraping.

Scraping a web page in python involves two process steps performed by different python packages — Selenium webdriver to get the html page source from a url query and BeautifulSoup html parser.

Fetch : (selenium) the raw html source code for the page from a url Extract :(beautifulsoup4) features from the source

Selenium webdriver

The python package selenium enables automation of web page interfaces using a webdriver. The webdriver behaves like a webbrowser but instead of a human interface has a scripting interface. Each browser offers their own webdriver — such as the Chrome Webdriver — which must be downloaded separately. The detailed instructions for getting started with selenium can be found on the package website.

Once the package is installed and the webdriver downloaded, retrieving a page source from a url only takes a few lines of code.

First a webdriver instance is created. Only one instance is needed and can be called repeatedly for multiple url page queries.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options ...

chrome_options = Options()

chrome_options.add_argument("--headless")

...

driver = webdriver.Chrome(executable_path=os.path.abspath('chromedriver'), options=chrome_options)

For those that use the PyCharm IDE, keeping the driver loaded and ready in the background at runtime can prevent the ability to stop the interpreter. The interpreter might hang and not shutdown or restart. To avoid this problem shutdown the webdriver after you are finished using it before you want to shutdown the interpreter using the .quit() command.

driver.quit()

The .get() method fetches the page source from the web as a (very lage) string and stores it to the .page_source property of the webdriver object.

driver.get(url)

pageStr = driver.page_source

Any communication over the web is slow compared to local runtime operations and limited by both the network connection and bottlenecks on the server and client side. For that reason any program that involves web communication interface should always minimize the number of fetch calls to the external server. When tested with a standard Singtel (local Singapore telcom provider) broadband connection the .get() call for a MyCareerFutures search query varies from 1.4 to 2.6 sec with average of 2.0 sec.

webdriver query page fetch time = 2.0 seconds

Because the time taken to perform operations on the server side is orders of magnitude faster than the communication step, the fetch time is practically invariant to the type of query whether it is a list of jobs or a single job. At the time of writing this post it’s not clear whether the site offers an option to increase the # of job cards displayed per query but if such an option was available that would be one way to speed up overall performance.

Parsing webpage html using BeautifulSoup

The motivation for fetching the html source code from the url is to extract fields on the page for analysis. Typical fields for analysis could be the # of job posts returned for a query, or any of the 11 fields available in a job profile mentioned earlier. For even the most basic web pages the raw html source code is not directly useful and some minimum parsing is needed in order to extract useful information for analysis. The html source code contains a mix of these structured fields and instructions to the browser for rendering the html for display to a human. The fields of interest represent a minority of the total information contained in the code so for many web pages the parsing process is like searching for a needle in a haystack.

snippet of html source code for a job profile

Parsing can be sub-divided into three steps :

Isolate : the html markers that uniquely identify the feature Navigate : the html source code extract the field contents Format : the contents into the structured for the database field

The basic pattern of an html and xml element is content … surrounded by html tags * indicated by the < *>…< */> characters at the beginning and end.

<tag>content</tag>

The contents between the tags can also contain subelements in a pattern that enables content to be nested in other more complex html elements and components.

<html>

<title>Product Manager Banking and Finance Jobs Singapore | MyCareersFuture Singapore</title>

<body>

<p> this is a great position for anyone looking to get experience in FinTech

</p>

... </body>

</html>

The tags are signposts that identify where specific type of information should be organized on the page. An example of a tag that identifies where to find the MCF Ref on a profile page is the html identifier : ”jobinfo__jobpostid — span”.

<span class="black-60 db f6 fw4 mv1" data-cy="jobinfo__jobpostid--span">MCF-2020-0229943</span>

The html tag class in this example is ‘span’. Other examples of tag classes are ‘div’ or ‘h1’. These are generic classes that apply to a wide range of fields on the page and the html identifier provides a more specific reference for that unique field.

The table here lists a few of the tag classes and identifiers for a selection of the 11 fields for a given job that can be found on the job card in a page listing of multiple job posts returned from a query.

html information for select job card fields

The easiest way to build such a table for each of the features of interest is to open up an instance of the page rendered on your browser in one window, and open up a text file of the source code and use CTRL+F to one-by-one identify the markers for each of the fields of interest.

Extracting one or two easily identifiable fields from a web pages the html element of interest could be performed using built-in python string or regular expression methods such as .find() directly on the page source string.

In order to build a database of records for multiple job posts, the section of html code for a specific job card must be readily isolated from the rest of the source code. The job card is not a single element but rather a collection of elements in a very long block of html code. While the beginning of the bloc is easy to identify, the end tag </ *> may be harder to identify because the same end tag is used for many other elements inside and outside of the job card. The only way to be sure is to keep track of the nested structure when tags have been closed out or a new one is opened up, similar to how we keep track of the order of operations of arithmatic using BODMAS.

While this may be do-able for someone with extensive text manipulation coding experience, it may require elaborate coding algorithm and could be intractable for someone with only basic familiarity with text manipulation.

BeautifulSoup

Fortunately the BeautifulSoup python package beautifulsoup4 overcomes this complexity by providing the programmer with the option to navigate the page as a nested tree structure of page elements as an alternative to dealing directly with the raw html page source string.

import bs4

...

pageSoup = bs4.BeautifulSoup(pageStr,"html5lib")

Instructions for developers on how to use BeautifulSoup to navigate through elements in an html source code is provided in BeautifulSoup documentation library. The html source code is converted first to a BeautifulSoup object which contains the html elements organized that can be queried and returned as Tag objects.

general illustration of how html element tags are contained within a BeautifulSoup page object

The .find_all() method returns the general class of html elements such as ‘div’ or ‘span’ as Tag objects. Both the Tag and the BeautifulSoup classes share the the .find_all() method (probably through inheritance) so the Tag object has the recursive property that it can also contain other Tag objects (subtags). That’s the basics of how to navigate the nested html tree. The page BeautifulSoup is the root node in this nested tree. After recursive calls working down the tree, eventually the final leaf Tag will not contain any other subtags and can only return its contents.

Isolating the job card

Coming back to the problem of identifying a specific job card — To return a job card from the query results, and with it all of the html information that it contains, we only need to know which class the job card element belongs to = ‘div’ and the unique html tag identifier = ‘job-card-’. For this example the identifier is contained in the ‘id’ property of the job card Tag object.

Since every job card follows this same pattern, we can generate a list of job cards by searching for any Tag element that matches these conditions and store them into a python list in one line of code using .find_all(), .has_attr() and the pythonic bracket ‘[]’ item accesor notation.

cardTag = 'job-card-'

...

cards = [y for y in [x for x in pageSoup.find_all('div')

if x.has_attr('id')] if self.cardTag in y['id']]

Extract the salary feature from a job card

The pattern for navigating and accessing content from elements within a card is the same pattern as identifying elements within the page. For the case of the salary field, it is contained in a ‘div’ element and it just so happens that the font identification ‘lh-solid’ suffices to uniquely identify the salary field within the card. This imprecise shortcut may work fine within the card but wouldn’t work in general on a search of the entire page since that font may show up in other parts of the page. Other fields can be extracted from the job card using a similar approach.

salary_divTag = 'lh-solid'

...

divTags = cardObj.find_all('div')

salaryTag = [y for y in [x for x in divTags if

x.has_attr('class')] if salary_divTag in y['class']][0]

The Tag class offers several methods and properties to return the element contents such as .text which return a text version of the html element including any html markers of sub-elements and .get_text() which returns only the human-readable contents and strips any html in the sub-elements. The two methods would return the same results for a leaf Tag that has no sub-elements.

Format the salary element contents into structured field : salaryHigh (int)

All elements are returned as string, so for any field that is intended to be stored in another data type such as integer, float or date, there is an additional formatting step required. In python this is accomplished using the built-in type conversion methods int() or float(). In a number of cases the raw text element cannot be directly converted since it may contain a mix of types. For example the contents of the salary element are in the form

“$6,500 to $9,500”

So a number of pre-processing formatting operations need to be performed — remove the $ signs, the ‘to’ in the middle and decide whether to return the low value, high value, or an average of the two. Also, the built-in python method int() does not automatically recognize number formats with commas ‘,’ so they also need to be removed before converting to integer. In this example the high value is selected using the built-in string method .split() and returned as an integer field salaryHigh.

salaryHighStr = salaryTag.text.split('to')[1] # $8,000 (str)

salaryHigh = int(salaryHighStr.replace(',', '').replace('$', '')) # 8000 (int)

Job title feature extraction using Regular Expression

To calculate an estimate the likelihood of a callback response (profile match score), some useful features are extracted from the available fields and those features are then used to calculate the qualification match %. The qualification match and rank are then combined to estimate the profile match.

Extract features : from the unstructed fields that can be fed into a model Model : the title score from a set of features Initialize : the first set of parameters to get started generating feedback

The nature of both the feature extraction and the scoring step to convert those features into the match % at this stage are black box processes. As a learning problem, feedback information from real callback responses is used to tune the mode to improve accuracy of predicting a callback response. A learning process can update the tuning parameters for a given model form, propose a model form and optimize the feature extraction process.

job screening process as a learning problem

Learning requires feedback and will take some time to generate the minimum number of callback data points. The current version of jobsearch has not had time to generate this volume yet and does not include a learning process. In the meantime, some intial naive model form needs to be setup to get started, even if it is initially a crude approximation.

Naive, interpretable title score model

A naive model should be concise and make as few assumptions as possible to accurately reflect the situation. One form to get started is a simple linear combination of features.

basic linear combination of features x(i) with coefficients alpha(i)

Another design principle in Data Science that is relevant for the model design is that:

(ceteris paribus) a model that is interpretable is preferred to one that is not.

Starting off with a complex, opaque model before you have a basic familiarity with the underlying process is asking for trouble. Applying the interpretability principle to this example — If the features are selected so that they can be interpreted meaningfully by the candidate, then the candidate could intialize the coefficient weights in an intial subjective assessment — the equivalent to test driving this scoring algorithm in MANUAL MODE. The principle is related to an adage from process automation industry:

If you don’t know how to control the machine in MAN, then you won’t be able to design a control system that works in AUTO

If at some point while in AUTO the process produces an unexpected result, there is a familiar default MAN mode version to revert back to for diagnostics and repair. Another motivation for starting with a MAN model version is that there is opportunity for learning experience to gain insights into the features in the input dataset while waiting for sufficient training data of callback responses to accumulate. With sufficient training data accumulated and learned insights into the input dynamics, the transition to apply machine learning and more sophisticated models is more likely to lead to a concrete results on the orginal problem of finding a match.

Introducing the candidate into the process of constructing and interpreting the model adds new constraints on the model form which could be relaxed for a less interpretable model (such as ANN). In particular, the candidate’s limited bandwidth sets limits on the number of features that the model applies to the score. Imagine managing coefficients for a model based on a million features.

So how many features is appropriate?

Given human reading rates ~ 200 words per minute, if we take a rough approximation that reading 1 cell consumes 1 word-equivalent of reading bandwidth when managing an NxM feature table, then if we set a maximum table read time of ≤1 hr, a feature table with 2 columns (an index and a coefficient) works out to :

maximum features ≤6,000.

Given that there is a large space of possible words and character combinations, a program based on language processing is expected to have a large number of features. A 6,000 features model wouldn’t be impossible to work with but it would be cumbersome. The point here is that the goal is to minimize the number of these features to only as many as is needed for the model to meet the expected performance. This should be a consideration for designing the feature selection process.

Position title features

Among the fields available, the unstructured fields — ’position title’ and the ‘description’ contain information with the highest potential for an accurate assessment of the candidate’s qualification match for the role.

position title

description

Unlike the salary field which was readily converted into a standard integer data type, unstructured text fields require additional processing step to interpret the characters and words into a form that is conducive to analysis (numbers). In comparison to the description field which is several paragraphs containing hundreds of words, the position title is a less complex introduction starting point for feature extraction.

Here is a position title example for a Senior Engineering role

Senior Engineer, Analog Design FE (AMG)(2020–5317–1B)

Data cleanup

Not everything in this string is a meaningful feature for determining a qualification match % score. Two goals in data cleanup is to remove junk and standardize the format.

Remove Junk : remove meaningless words and characters ex )( — , Standardize Format : all lower case, only alphabet characters a-z

Junk is any meaningless set of characters or words that do not provide any information value for assessing the qualification match %

There are two general forms of junk — generic patterns and specific user-determined tags. While extra characters like punctuation ‘,’ or parenthesis () may be helpful if the way they are used is standardized across all of the job posts, unfortunately that isn’t the case so its better to remove them and focus on the alphabet characters a-z. A similar argument for numbers. Case sensitivity is less straightforward decision. Acronymns that are in all caps such as IHPC (A*STAR Institute for High Performance Computing) have high information value compared to their lowercase equivalent ihpc, but the downside is that this would add redundancy for the same features for : Data Scientist, data scientist and Data scientist. It is possible to distinguish between these two circumstances using Regular Expression, but for the first pass everything is set to all lower case.

User defined junk tags

The candidate can also supply a library set of junk words and character combinations and add to this library over time. Examples of position title junk words are numerous such as

contract 1 year

sgunitedjobs

progression benefits

ref

While some of these tags tell something about the role that may interest the candidate, they probably have little value in predicting whether the hiring manager will extend a callback.

Cleaning up strings in python

Two python methods which are useful for performing these cleanup operations are the built-in str.replace() and re.sub(). re is the python package implementation of Regular Expression, a standard format for string operations. It is a syntax for expressing a wide range of character patterns and used in natural language processing algorithms. A few examples :

remove ‘sgunitedjobs’ from a string using str.replace()

tag = 'sgunitedjobs'

strValue = strValue.replace(' '+tag+' ',' ')

drop all non-alphabet characters using re.sub() and change to all lower case using str.lower()

# keep only alphabet characters

strValue = re.sub('[^a-zA-Z ]+','',strValue)

# lower case only

strValue = strValue.lower()

The result after applying these and other similar operations is a clean title after removing junk and standardizing the format :

senior engineer analog design fe amgb

Remove rank

The next step is to remove the rank words, in this case ‘senior’, from the title. Examples of rank words are ‘senior’ ‘head’ ‘owner’. Below is a list of sample rank words listed in order of their relative predictive power on the rank proxy (within-industry percentile) using the scikit-learn LogisticRegression model. The cofficients from the regression can be interpreted as log-probability so values further than one provide more discriminating power. A smaller, more negative coefficient is more predictive of a lower rank and a larger, more positive value is more predictive of a higher rank. For some people outside of Singapore the placement of ‘executive’ as a dimunitive rank may seem in err but within the context that these titles are all roles >$6,000 this is close to how the word is used relative to these other rank titles in that group.

rank words (y-axis) vs relative predictive power for within-industry salary percentile (x-axis)

The rational for removing these words is that this information is redundant since the within-industry percentile provides a more accurate and consistient measure of rank. Relative to the salary information, there is more ambiguity from these words since they are inherently subjective and context-dependent within a particular firm. This reduces the total size of the feature space (senior engineer, head engineer, vp engineer → engineer) since almost every position includes a rank word and there are at least a dozen common rank words in the dataset. Fortunately the number of these words is small and they are easy to identify so they can be treated like another kind of junk word with a user-supplied tag library and removed using the same python methods.

Working with bigrams as string features using scikit-learn CountVectorizer

At this point the candidate could generate a library of cleaned, deranked titles by running these operations on a set of job posts and then manually assign subjective coefficient scores one-by-one to each of these titles, apply a score using the linear combination of the coefficients and use that as an MVP (minimum-viable-product) operational application. This could be implemented using a simple lookup operation using the title library.

This method however would quickly blow up because it doesn’t scale well. At a rate of 2,000 new posts per week, the number of posts would reach the target feature limit of 6,000 in ~ 3 weeks. As the database grows and new posts are added, the user has to continuously keep updating the tag library with new coefficient scores. While some of the titles might be duplicates and thus the number of features may be slightly < # of posts, the problem is that these titles are still quite unique and not very generalizable to some new set of posts and the # of features is likely to grow appx linearly with the # of posts.

One way of preserving the information but generalizing the features is to map from a list of titles to a list of n-grams. An n-gram is a combination of n-adjacent words. A single word is n=1, a bigram is n=2, etc.. An example of a bigram is ‘data scientist’.

For any string with number of words N> 1, the number of bigrams B in the word is

B=N-1

There area total of 4 unique bigrams in the deranked title for the Engineer example

bigrams = ['engineer analog', 'analog design', 'design fe', 'fe amgb']

Why apply an operation that increases the number of features per record when the motivation was to decrease them?

The reason is because many of these job titles share a common set of a few popular bigrams. A bigram is a common shorthand form for a job role ‘data scientist’ ‘process engineer’ ‘operations manager’. While the distribution does have long tail, many are repeated and occur in mutiple job profiles. From a set of >9,000 job posts there are < 700 unique bigrams and the top 90 most frequent bigrams occur in 50% of the job posts and the top 330 occur in 80% of the posts.

frequency (orange, left x-axis) and cumulative (grey, right x-axis) bigram distribution

More importantly, as new records get added, it is likely that they will come from the same distribution and share many of these same popular bigrams from the original set. If this is the case then the number of bigrams should increase sublinearly as the database increases, rather than linearly, a substantial improvement and more managable for the candidate. That means a smaller set of bigrams to update coefficients, on the order of 1,000-2,000, well within our target of <6,000 features to manage.

It is easy to generate a set of bigrams from a list of strings in python from a few lines of code using the CountVectorizer class from the general purpose machine learning package scikit-learn.

In the code example below titles is a pandas Series object list of deranked titles as string, X_v is a sparse bigram feature matrix and feature_names is the list of unique bigrams that corresponds to the columns in X_v.

from sklearn.feature_extraction.text import CountVectorizer

...

vect = CountVectorizer(min_df=5, ngram_range=(2, 2), analyzer='word').fit(titles)

X_v = vect.transform(titles)

feature_names = np.array(vect.get_feature_names())

CountVectorizer generates an NxM feature matrix X_v where the N-rows correspond to the list of strings and the M-columns are the n-grams. Each cell value i,j is a count for the number of instance occurances of the n-gram j for the string in row i.

CountVectorizer matrix X_v image credits to Thalia Wang https://medium.com/@wenxuan0923/feature-extraction-from-text-using-countvectorizer-tfidfvectorizer-9f74f38f86cc

The title score can be computed using matrix multiplication * of the NxM feature matrix by a Mx1 vector of the feature coefficients from the user-defined n-gram tag library.

Title scores = X_v * feature_scores

The sparse matrix must be converted to a standard matrix form using .todense() before applying matrix multiplication.

import numpy as np

...

scores = np.matmul(X_v.todense(),feature_scores.transpose())

Building the bigram tag library

In this application, three coefficient values are chosen -1 (not qualified) +1 (qualified) and 0 (partial qualification or not enough information to determine). Between bigram tag library updates any bigram generated from a new set of job posts that wasn’t found in the libary is assigned a default score of 0.

bigram tag library with the user-defined coefficient in column (score)

Inbox : Concentrate on what matters

The match score can now be computed from the match formula using the rank and the title score. The match score can then be used as a filter to screen the posts that the user will then consider applying for.

For a baseline reference of 9,000 posts from 2020–11–23 applying a filter of match ≥0 is 10% selective, and the selectivity increases by 10-fold with every 1 point score increase. So on a first pass the program meets the specifications and has enough flexibiltiy for tuning and future learning.

Now instead of a list of 9,000 mostly unrelated jobs to sift through, the candidate starts with this concentrated set of screened job posts and can focus their attention on the high value-added areas that the career professionals are recommending.