Systems and methods to develop training set of data based on resume corpus

Systems, methods, and non-transitory computer readable media are configured to acquire a resume corpus. The resume corpus is processed to generate resume tokens. A machine learning model is trained based on the resume tokens. The machine learning model is applied to recommend a job classification based on evaluation data.

FIELD OF THE INVENTION

The present technology relates to the field of machine learning. More particularly, the present technology relates to techniques for training a word vector model based on a resume corpus.

BACKGROUND

Recruiters can play a primary role in helping organizations locate job candidates. In some cases, a recruiter can proactively seek job candidates for the organization. In other cases, job candidates can initiate contact with an organization through a recruiter of the organization. The process to assess job candidates often can be initiated through electronic receipt by the organization of a resume of a job candidate. An organization can receive large volumes of resumes. The sheer number of resumes received by such an organization can create challenges for the recruiter in vetting the resumes to identify job candidates suited to the organization.

SUMMARY

Various embodiments of the present technology can include systems, methods, and non-transitory computer readable media configured to acquire a resume corpus. The resume corpus is processed to generate resume tokens. A machine learning model is trained based on the resume tokens. The machine learning model is applied to recommend a job classification based on evaluation data.

In an embodiment, the resume corpus is based on textual information from a plurality of resumes.

In an embodiment, the resume tokens include one or more unigrams and one or more bigrams.

In an embodiment, processing of the resume corpus comprises: removing stop words from the resume corpus.

In an embodiment, the stop words include at least one of pronouns, prepositions, articles, and conjunctions.

In an embodiment, processing of the resume corpus comprises: modifying capitalized letters of words in the resume corpus to have lowercase letters.

In an embodiment, processing of the resume corpus comprises: determining a value based on a number of instances in which a first word and a second word co-occur and a number of instances in which the first word and the second word occur individually.

In an embodiment, processing of the resume corpus comprises: generating a whitelist of bigrams constituting job titles parsed from the resume corpus.

In an embodiment, processing of the resume corpus further comprises: including a bigram in the whitelist based on satisfaction of a threshold appearance value relating to the bigram.

In an embodiment, the job classification includes a job title or a job pipeline.

DETAILED DESCRIPTION

Creating Dictionary for Job Recommendation

As mentioned, recruiters can play a primary role in helping organizations locate job candidates. In some cases, a recruiter can proactively seek job candidates for the organization. In other cases, job candidates can initiate contact with an organization through a recruiter of the organization. The process to assess job candidates often can be initiated through electronic receipt by the organization of a resume of a job candidate. Certain organizations can receive large volumes of resumes. The sheer number of resumes received by such organizations can create challenges for recruiters in vetting the resumes to identify suitable job candidates.

Textual data can be used to train machine learning models for a variety of applications, including job candidate identification. Conventional training data can be, for example, raw text associated with Wikipedia or from other source of information. Many disadvantages are associated with the use of such conventional training data in the context of job candidate identification. For example, because conventional training data is general, it can introduce ambiguities based on words that may have one meaning with respect to job candidate identification and other meanings with respect to other endeavors. As another example, conventional training data may not be well structured in that closely related words may not appear in close proximity to one another. As yet another example, the conventional training data can include figurative language, sarcasm, emotional language, or the like that can mislead about the meaning of the data. As a result, the efficacy of a machine learning model developed from conventional training data can be impacted by these disadvantages.

An improved approach rooted in computer technology overcomes the foregoing and other disadvantages associated with conventional approaches specifically arising in the realm of computer technology. Systems, methods, and computer readable media of the present technology can acquire a resume corpus based on a large number of resumes. Stop words from the resume corpus can be removed from the resume corpus. Capitalized words in the resume corpus can be changed to lowercase words to facilitate deduplication of the resume corpus. Bigrams can be identified according to one or more techniques. In one example technique, bigrams can be identified based on a TF-IDF technique involving frequency of co-occurring words. In another example technique, a whitelist of bigrams can be curatively determined from job titles in resumes. Resume tokens including unigrams and the identified bigrams can constitute a training set of data to train a machine learning model. The machine learning model can be based on a technique that converts the resume tokens into vector representations in a vector space based on meaning of the tokens. One example machine learning model is word2vec. The machine learning model can be used for recommendations of job classification, such as job titles and job pipelines. For example, the machine learning model can be used to determine a suitable job title for a job candidate based on profile information of the candidate. As another example, the machine learning model can be used to determine a degree to which one or more resumes match search terms selected by a recruiter that relate to a job pipeline. More details regarding the present technology are described herein.

FIG. 1illustrates an example system100including an example job classification recommendation module102configured to recommend job classifications for persons, such as recruiters, for an organization (e.g., a technology company), according to an embodiment of the present technology. Job classifications, as used herein, can refer to terms that span a spectrum between coarse descriptors to fine grained descriptors associated with or otherwise indicative of a job position, responsibility, role, category, or other scope of job. As used herein, “job title” may be used as a relatively coarse descriptor of a job classification and “job pipeline” may be used as a relatively fine grained descriptor of a job classification to illustrate the effectiveness of the job classification recommendation module102across a spectrum of both general and specific job classification types. In some embodiments, an express reference to a “job title” herein can also apply to a job pipeline and an express reference to a “job pipeline” herein can also apply to a job title. An organization can be any entity, such as a company, an establishment, a non-profit, a business, etc. The organization can be of any type or in any industry, such as aerospace and defense, agriculture, automotive, chemicals, construction, consumer goods and services, energy, financial services, firearms, food and beverage, health care, information and technology (e.g., software, hardware, etc.), real estate, manufacturing, mining and drilling, pharmaceuticals and biotechnology, publishing, telecommunications, transportation, etc. While a technology company or related job classifications may be exemplarily discussed in certain contexts for ease of explanation herein, an organization of any industry type or endeavor can be applicable to the present technology. For example, the present technology can be applied to any other type of organization by tailoring the training of a machine learning model with a training set of data that is relevant to the type of organization and its recruiting strategy. Further, the application of the machine learning model described in relation to the job classification recommendation module102can be applied to other purposes apart from job classification recommendations.

The job classification recommendation module102can include a model creation module104, a job classification determination module106, and a presentation module108. The components (e.g., modules, elements, steps, blocks, etc.) shown in this figure and all figures herein are exemplary only, and other implementations may include additional, fewer, integrated, or different components. Some components may not be shown so as not to obscure relevant details. In various embodiments, one or more of the functionalities described in connection with the job classification recommendation module102can be implemented in any suitable combinations.

The model creation module104can acquire resumes (or CVs) and extract textual data from the resumes. The textual data can constitute a resume corpus. The resume corpus can be processed to generate a training set of data. For example, stop words can be identified and removed from the resume corpus. As another example, bigrams can be identified in the resume corpus. Other processing can be performed on the resume corpus. Resume tokens from the resume corpus can constitute unigrams and the identified bigrams. The resume tokens can constitute a training set of data to be provided to a machine learning model. The machine learning model can create vector representations of the resume tokens in a vector space. Use of the resume tokens instead of raw conventional data as the training set to train the machine learning model poses advantages in the use of the machine learning model for effective recommendation of job classifications. The model creation module104is discussed in more detail herein.

The job classification determination module106can apply the machine learning model developed by the training set of data to recommend job classifications. In some embodiments, the machine learning model can facilitate a recommendation of an appropriate job title based on evaluation data, such as profile data of a job candidate. In this regard, the profile information can be maintained by a social networking system. Based on the machine learning model, job titles of interest can be converted to vector representations constituting anchor points in the vector space. Various profile information types of the job candidate, such as job titles, educational majors, and educational degrees, can be processed by application of the profile information types to the model. The machine learning model, as discussed in more detail herein, can convert terms associated with profile information into vector representations in a vector space based on meaning of the terms. A vector representation for each term of a profile information type can be identified. For each vector representation of each term of a profile information type, a nearest anchor point can be identified subject to a threshold distance value condition. Based on a hierarchical rule, an anchor point can be chosen from the identified anchor points associated with terms of the profile information types. The chosen anchor point can represent a job title matched to the job candidate.

In some embodiments, the machine learning model can be applied to facilitate a determination as to a degree to which one or more resumes match evaluation data constituting search terms selected by a recruiter relating to a job pipeline. In this regard, the machine learning model, as discussed in more detail herein, can convert the search terms into vector representations in a vector space based on meaning of the terms. A set of search terms used by a plurality of recruiters can be represented in the vector space as a set of keywords (or anchor points). When a recruiter wishes to perform one or more searches on a set of resumes, each resume can be converted into an array of values representing a frequency of unique keywords by determining, for each identified chunk of terms in the resume, a nearest keyword. Each array of values representing a frequency of unique keywords can be normalized to reflect the relative importance of the keywords associated with array. Arrays of values representing a frequency of unique keywords for the set of resumes can be rows in a resume matrix. Search terms of the recruiter for each search to be performed can be expressed as an array of values representing a frequency of search terms associated with the search. Arrays of values representing a frequency of search terms associated with various searches can be columns in a search term matrix. A matrix multiplication can be performed for the resume matrix and the search term matrix to generate a score matrix. The resume scores reflected in the score matrix can determine one or more resumes most related to each search as well as one or more searches to which each resume is most related.

The presentation module108, through a suitable user interface, can allow an entity, such a recruiter, to use the job classification recommendation module102to identify suitable job candidates. For example, the presentation module108, through the user interface, can present matches between job candidates and suitable job titles, subject to applicable privacy settings. In some embodiments, the presentation module108can provide, via the user interface, the presentation to users who are employees of an organization to whom the job candidates are connected in a social network supported by a social networking system. The users can include recruiters of or for the organization. The presentation module108also can provide options for users to refer the job candidates to the organization.

As another example, the presentation module108, through a suitable user interface, can prompt a user to provide one or more search terms for each search relating to a job pipeline to be performed against a set of resumes. In response to provision of the set of resumes and search terms, the presentation module108, through the user interface, can present, for each search, a list of resumes relevant to the search. In addition, the presentation module108, through the user interface, can present, for each resume, a list of searches that are most relevant to the resume. A list can be sorted based on applicable resume scores.

In some embodiments, the job classification recommendation module102can be implemented, in part or in whole, as software, hardware, or any combination thereof. In general, a module as discussed herein can be associated with software, hardware, or any combination thereof. In some implementations, one or more functions, tasks, and/or operations of modules can be carried out or performed by software routines, software processes, hardware, and/or any combination thereof. In some cases, the job classification recommendation module102can be, in part or in whole, implemented as software running on one or more computing devices or systems, such as on a server or a client computing device. For example, the job classification recommendation module102can be, in part or in whole, implemented within or configured to operate in conjunction or be integrated with a social networking system (or service), such as a social networking system630ofFIG. 6. As another example, the job classification recommendation module102can be implemented as or within a dedicated application (e.g., app), a program, or an applet running on a user computing device or client computing system. In some instances, the job classification recommendation module102can be, in part or in whole, implemented within or configured to operate in conjunction or be integrated with client computing device, such as a user device610ofFIG. 6. It should be understood that many variations are possible.

A data store118can be configured to store and maintain various types of data, such as the data relating to support of and operation of the job classification recommendation module102. The data can include data relating to, for example, a resume corpus, stop words, bigrams from the resume corpus, unigrams from the resume corpus, resume tokens, a machine learning model to create vector representations of resume tokens, etc. The data store118also can maintain other information associated with a social networking system. The information associated with the social networking system can include data about users, social connections, social interactions, locations, geo-fenced areas, maps, places, events, groups, posts, communications, content, account settings, privacy settings, and a social graph. The social graph can reflect all entities of the social networking system and their interactions. As shown in the example system100, the job classification recommendation module102can be configured to communicate and/or operate with the data store118.

FIG. 2illustrates an example model creation module202, according to an embodiment of the present technology. In some embodiments, the model creation module104ofFIG. 1can be implemented with the model creation module202. The model creation module202can include a resume corpus processing module204and a token vectorization module206.

The resume corpus processing module204can determine a resume corpus constituting textual data from a plurality of resumes. The resume corpus processing module204can remove certain terms from the resume corpus. In some embodiments, the removed terms can include stop words. The resume corpus processing module204also can identify bigrams from the resume corpus. The resume corpus processing module204can perform other processing on the resume corpus. Resume tokens including unigrams and the identified bigrams from the resume corpus can constitute a training set of data to train a machine learning model. The resume corpus processing module204is described in more detail herein.

The token vectorization module206can generate a machine learning model to create a vector space of vector representations of the resume tokens of the training set. The machine learning model generated by the token vectorization module206can be trained using any suitable technique (or algorithm) that can create a vector space of vector representations of resume tokens (e.g., unigrams and bigrams) from resumes based on semantic meaning of the terms. In this regard, for terms that are relatively close in meaning, the machine learning model can create vector representations of the resume tokens that are relatively close to one another in the vector space. Likewise, for terms that are relatively far in meaning, the machine learning model can create vector representations of the resume tokens that are relatively far from one another in the vector space. Resume tokens with similar or identical meanings are clustered together. The vector space can represent a dictionary of terms. As just one illustration, the resume corpus can be based on approximately one million resumes and the vector space can represent a dictionary of approximately eight million terms taken from the resumes, depending on the content of the resumes. In some embodiments, the technique to create a vector space of vector representations of terms can be based at least in part on a word2vec technique.

FIG. 3illustrates an example resume corpus processing module302, according to an embodiment of the present technology. The resume corpus processing module302can acquire and process resume data to train a machine learning model that can determine job classification recommendations. The use of resume data to train a machine learning model to determine job classification recommendations can pose many advantages over conventional use of other generalized, raw (or unprocessed) training data. Because the processed resume data pertain specifically to job classification determinations, the machine learning model can avoid ambiguities based on words that may have one meaning with respect to job classification determinations yet other meanings with respect to other endeavors. For example, the appearance of the word “bug” could have one meaning in the context of a resume relating to a certain job classification (e.g., software engineering) and could have other, different meanings in other generalized contexts. Further, the processed resume data is well structured in that closely related words are likely to appear in close proximity to one another, further enhancing training of the machine learning model. For example, resume data is often characterized by a stated job classification immediately followed an explanation of or elaboration about the job classification. Such structured data is well suited to the development of an accurate machine learning model. Further still, the processed resume data likely does not contain figurative language, sarcasm, emotional language, or the like that can mislead about the meaning of the data. In this regard, the omission of such misleading language can enhance the accuracy of the machine learning model for job classification determinations.

In some embodiments, the resume corpus processing module204ofFIG. 2can be implemented with the resume corpus processing module302. The resume corpus processing module302can include a resume collection module304, a stop words and lowercase module306, a co-occurrence bigram determination module308, and a bigram whitelist determination module310.

The resume collection module304can acquire a number of resumes to generate a training set of data to train a machine learning model. The number of resumes can be any suitable number of resumes, such as, for example, one million resumes. In some embodiments, a larger number of resumes or a smaller number of resumes can be used. In some instances, the resumes can include resumes received by an organization. For example, the resumes can include resumes of employees of the organization.

The stop words and lowercase module306can identify stop words in the resume corpus and remove the stop words from the resume corpus. In general, stop words can refer to common words or semantically insignificant words that may be excluded from natural language processing techniques. In various embodiments, the stop words and lowercase module306can identify and remove one or more stop words or type of stop words in the resume corpus. Types of stop words can include, for example, pronouns, prepositions, articles, conjunctions, and the like. In addition, the stop words and lowercase module306can modify some or all words in the resume corpus so that capitalized letters of the words are changed to lowercase letters. In some embodiments, deduplication of words in the resume corpus is performed. The use of lowercase letters instead of capitalized letters in words can facilitate such deduplication.

The co-occurrence bigram determination module308can determine bigrams from the resume corpus. In some embodiments, the bigrams can be determined using a TF-IDF technique based on word co-occurrence. In some embodiments, determination of a bigram can be based on satisfaction of a threshold. The threshold can relate to whether a first word (e.g., “software”) and a second word (e.g., “engineer”), which together constitute a potential bigram (e.g., “software engineer”), co-occur (i.e., occur consecutively) at a sufficiently high frequency value in the resume corpus (or another corpus of data) in relation to whether the first word and the second word do not co-occur (i.e., occur non-consecutively) at a sufficiently low frequency value in the resume corpus.

In some embodiments, the determination of a bigram based on a first word and a second word that co-occur can be based on a ratio of a number of instances in which the first word and the second word co-occur divided by a product of a number of instances in which the first word occurs individually and a number of instances in which the second word occurs individually. The ratio can be compared to a threshold value to determine the existence of a bigram. If the ratio based on the first word and the second word does not satisfy the threshold value, the first word and the second word are not deemed to constitute a bigram. Similarly, if the ratio based on the first word and the second word does satisfy the threshold value, the first word and the second word are deemed to constitute a bigram. The threshold value is configurable. In some embodiments, a higher threshold value can be selected when a higher confidence level regarding the existence of a bigram is required. In some embodiments, a lower threshold value can be selected when a lower confidence level regarding the existence of a bigram is acceptable.

The bigram whitelist determination module310can determine bigrams from a whitelist reflecting job titles taken from the resumes. The bigram whitelist determination module310can identify bigrams that have not been identified by the co-occurrence bigram determination module308. For example, based on certain threshold values, the co-occurrence bigram determination module308might fail to identify “software developer” as a bigram. Accordingly, the bigram whitelist determination module310can parse the resumes to determine job titles from the resumes or can receive job titles from resumes parsed by a third party system. The job titles from the resumes can be identified as such based on the typical structure and organization of the conventional format of standard resumes. The parsing can be done automatically or manually. Bigrams associated with the job titles can be identified in this manner for potential inclusion in a whitelist of bigrams.

In some cases, the parsing of a resume can result in the identification of a job title having three or more words. In such cases, the bigram whitelist determination module310can associate pairs of consecutive words as potential bigrams. For example, for a job title having three words (e.g., “senior software developer”) that has been parsed from a resume, the bigram whitelist determination module310can associate the first word and the second word as a bigram (i.e., “senior software”) for potential inclusion in the whitelist. Likewise, the bigram whitelist determination module310can associate the second word and the third word as another bigram (i.e., “software developer”) for potential inclusion in the whitelist. In some embodiments, a number of appearances in the resume corpus of a bigram for potential inclusion in the whitelist must satisfy a threshold appearance value before the bigram can be included in the whitelist. When the number of appearances of the bigram does not satisfy the threshold appearance value, the bigram can be excluded from the whitelist. For instance, the threshold appearance value can be, for example, 20 or some other suitable value.

In some embodiments, the resume corpus processing module302can perform other types of processing to optimize unigrams and bigrams as resume tokens as a training set of data. In some instances, additional techniques to determine bigrams can be implemented in view of the semantic importance and specificity of bigrams, which can facilitate more optimal training of a machine learning model. In some instances, certain unigrams or bigrams can be eliminated from further consideration to account for anomalies or mistakes in the resume corpus or to enhance the quality of the data provided for training. The unigrams and the bigrams resulting from the processing performed by the resume corpus processing module302are resume tokens that constitute a training set of data to be provided to the token vectorization module206to train a machine learning model.

FIG. 4illustrates a first example method400to generate a training set of data to train a machine learning model to recommend job classifications, according to an embodiment of the present technology. It should be appreciated that there can be additional, fewer, or alternative steps performed in similar or alternative orders, or in parallel, in accordance with the various embodiments and features discussed herein unless otherwise stated.

At block402, the method400can acquire a resume corpus. At block404, the method400can process the resume corpus to generate resume tokens. At block406, the method400can train a machine learning model based on the resume tokens. At block408, the method400can apply the machine learning model to recommend a job classification based on evaluation data. Other suitable techniques that incorporate various features and embodiments of the present technology are possible.

FIG. 5illustrates an example method500to process a resume corpus to generate resume tokens to train a machine learning model, according to an embodiment of the present technology. It should be appreciated that there can be additional, fewer, or alternative steps performed in similar or alternative orders, or in parallel, in accordance with the various embodiments and features discussed herein unless otherwise stated.

At block502, the method500can remove stop words from a resume corpus. At block504, the method500can modify capitalized letters of words in the resume corpus to have lowercase letters. At block506, the method500can determine a value based on a number of instances in which a first word and a second word co-occur and a number of instances in which the first word and the second word occur individually. At block508, the method500can generate a whitelist of bigrams constituting job titles parsed from the resume corpus. Other suitable techniques that incorporate various features and embodiments of the present technology are possible.

Social Networking System—Example Implementation

The user device610comprises one or more computing devices that can receive input from a user and transmit and receive data via the network655. In one embodiment, the user device610is a conventional computer system executing, for example, a Microsoft Windows compatible operating system (OS), Apple OS X, and/or a Linux distribution. In another embodiment, the user device610can be a device having computer functionality, such as a smart-phone, a tablet, a personal digital assistant (PDA), a mobile telephone, etc. The user device610is configured to communicate via the network655. The user device610can execute an application, for example, a browser application that allows a user of the user device610to interact with the social networking system630. In another embodiment, the user device610interacts with the social networking system630through an application programming interface (API) provided by the native operating system of the user device610, such as iOS and ANDROID. The user device610is configured to communicate with the external system620and the social networking system630via the network655, which may comprise any combination of local area and/or wide area networks, using wired and/or wireless communication systems.

The external system620includes one or more web servers that include one or more web pages622a,622b, which are communicated to the user device610using the network655. The external system620is separate from the social networking system630. For example, the external system620is associated with a first domain, while the social networking system630is associated with a separate social networking domain. Web pages622a,622b, included in the external system620, comprise markup language documents614identifying content and including instructions specifying formatting or presentation of the identified content.

In some embodiments, the social networking system630can include a job classification recommendation module646. The job classification recommendation module646can be implemented with the job classification recommendation module102, as discussed in more detail herein. In some embodiments, one or more functionalities of the job classification recommendation module646can be implemented in the user device610.

Hardware Implementation