SkillBERT: “Skilling” the BERT to classify skills!

1 INTRODUCTION . Competency group can be thought of as a group of similar skills required for success in a job . For example , skills such as Apache Hadoop , Apache Pig represent competency in Big Data analysis while HTML , Javascript are part of Front-end competency . Classification of skills into the right competency groups can help in gauging candidate ’ s job interest and automation of the recruitment process . Recently , several contextual word embedding models have been explored on various domain-specific datasets but no work has been done on exploring those models on job-skill specific datasets . Fields like medical and law have already explored these models in their respective domains . Lee et al . ( 2019 ) in their BioBERT model trained the BERT model on a large biomedical corpus . They found that without changing the architecture too much across tasks , BioBERT beats BERT and previous state-of-the-art models in several biomedical text mining tasks by a large difference . Alsentzer et al . ( 2019 ) trained publicly released BERT-Base and BioBERT-finetuned models on clinical notes and discharge summaries . They have shown that embeddings formed are superior to a general domain or BioBERT specific embeddings for two well established clinical NER tasks and one medical natural language inference task ( i2b2 2010 ( Uzuner et al. , 2011 ) , i2b2 2012 ( Sun et al. , 2013a ; b ) ) , and MedNLI ( Romanov & Shivade , 2018 ) ) . 1https : //www.dropbox.com/s/wcg8kbq5btl4gm0/code_data_pickle_files.zip ? dl=0 ! Beltagy et al . ( 2019 ) in their model SciBERT leveraged unsupervised pretraining of a BERT based model on a large multi-domain corpus of scientific publications . SciBERT significantly outperformed BERT-Base and achieves better results on tasks like sequence tagging , sentence classification , and dependency parsing , even compared to some reported BioBERT results on biomedical tasks . Similarly , Elwany et al . ( 2019 ) in their work has shown the improvement in results on fine-tuning the BERT model on legal domain-specific corpora . They concluded that fine-tuning BERT gives the best performance and reduces the need for a more sophisticated architecture and/or features . In this paper , we are proposing a multi-label competency group classifier , which primarily leverages : SkillBERT , which uses BERT architecture and is trained on the job-skill data from scratch to generate embeddings for skills . These embeddings are used to create several similarity-based features to capture the association between skills and group . We have also engineered features through clustering algorithms like spectral clustering on embeddings to attach cluster labels to skills . All these features along with SkillBERT embeddings are used in the final classifier to achieve the best possible classification accuracy . 2 METHODOLOGY . As no prior benchmark related to job-skill classification is available , we manually assigned each skill in our dataset to one or more competency groups with the help of the respective domain experts to create training data . We experimented with three different models : pre-trained BERT , Word2vec , and SkillBERT to generate word embeddings . Word2vec and SkillBERT were trained from scratch on our skill dataset . We created some similarity-based and cluster-based features on top of these embeddings . Except for these features , some frequency-based and group-based features were also generated . A detailed explanation of all the steps is mentioned in the next sections . The details of dataset design and feature engineering used for model creation are given in the next sections . 2.1 TRAINING DATA CREATION . Our approach uses a multi-label classification model to predict competency groups for a skill . However , as no prior competency group tagging was available for existing skills , we had to manually assign labels for training data creation . For this task , the skill dataset is taken from our organization ’ s database which contains 700,000 job requisitions and 2,997 unique skills . The competency groups were created in consultation with domain experts across all major sectors . Currently , there exists 40 competency groups in our data representing all major industries . Also within a competency group , we have classified a skill as core or fringe . For example , in marketing competency group , digital marketing is a core skill while creativity is a fringe skill . Once training data is created , our job is to classify a new skill into these 40 competency groups . Some skills can belong to more than one category also . For such cases , a skill will have representation in multiple groups . Figure 1 shows an overview of the datasets used in this step . 2.2 FEATURE ENGINEERING . For feature creation , we have experimented with Word2vec and BERT to generate skill embeddings . By leveraging these skill embeddings we created similarity-based features as well . We also used clustering on generated embeddings to create cluster-based features . As multiple clustering algorithms are available in the literature , we evaluated the most popular clustering algorithms – K-means ( Lloyd , 1982 ) and spectral clustering for experimentation . We have done extensive feature engineering to capture information at skill level , group level , and skill-group combination level . The details of features designed for experiments are given below . 2.2.1 EMBEDDING FEATURES . Traditionally , n-gram based algorithms were used to extract information from text . However , these methods completely ignore the context surrounding a word . Hence , we have experimented with Word2vec and BERT based architecture to learn embeddings of skills present in training data . The details of how we have leveraged them in our problem domain are given below . Word2vec is one of the most popular techniques to learn word embeddings using a neural network . It uses a shallow , two-layer neural network to generate n-dimensional embedding for words . To use the Word2vec model on requisition data , we extracted skills from job requisitions and constructed a single document . Each sentence of this document represents the skills present in one requisition . As a requisition can have multiple skills , we created a 2-dimensional list , where the outer dimension specifies the sentence and inner dimension corresponds to the skills in that sentence . E.g . if there are two job requisitions called req1 and req2 and their corresponding skills are `` Java , J2EE '' and `` Logistic regression , Data visualization , NLP '' then outer index 0 corresponds to req1 and outer index 1 corresponds to req2 . Index 0,0 will refer to Java and Index 0,1 will refer to J2EE and so on . Also before feeding this data for training lowercasing of words , stop word removal and stemming was performed as part of preprocessing . A total of more than 700,000 requisitions were used for model training . We have used embeddings of size 30 which was decided after evaluating model performance on different embedding sizes . BERT Bidirectional Encoder Representations from Transformers , is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers . Pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for tasks such as question answering , next sentence prediction , etc . Similar to Word2vec , BERT can also be used to extract fixed-length embeddings of words and sentences , which can further be used as features for downstream tasks like classification . But unlike fixed embedding produced by Word2vec , BERT will generate different embedding for an input word based on its left and right context . BERT has shown performance improvement for many natural language processing tasks . However , it has been minimally explored on the job-skill database . Hence , we leveraged BERT architecture on skill data to train the SkillBERT model . We have leveraged ml.p2.xlarge type 12 GIB GPU memory 1xK80 GPU available on the AWS cloud for SkillBERT training and it took us around 72 hours to completely train it on our dataset . In the next section , we have given the details of training BERT on skill corpus . Training : For training BERT , we used the same corpus as used for Word2vec training and experimented with hyperparameters like learning rate and maximum sequence length . For the learning rate , we used 0.1 , 0.05 , and 0.01 and finalised 0.01 . For maximum sequence length , we used 64 , 128 , 180 and finalised 128 . We could not perform extensive hyperparameter tuning due to hardware limitations . Once the training is finished , we extract the last hidden layer output of size 768 and further reduce the embedding size to 128 to decrease the training time of our final model discussed in the experiment section . For the dimensionality reduction of embedding , we did experiments with embeddings of size 32 , 64 , 128 and 256 . As shown in Appendix Table 6 , the best results were obtained using embedding of size 128 . To make sure information from all the 768 dimensions is leveraged , we trained a 2-layer neural network classifier using SkillBERT embeddings as an independent feature and competency group as a dependent variable . Out of the 2,997 skills , 80 % were used for training and the rest of the 20 % were used for the validation . This model generates the probability values of a skill belonging to each of the 40 competency groups and was used as a feature in the final model at skill and competency group combination level . We have referred this feature as `` bert-prob '' in the rest of the sections . Figure 2 represents the architecture of the model used for getting these probabilities . 2.2.2 SIMILARITY-BASED FEATURES . By leveraging skill embeddings generated using embedding techniques , similarity-based features were created to capture the association between a group and skill . The details of those are given below . Similarity from competency group : In competency group data , the name of each competency group is also present as a skill . We created a similarity score feature measuring the cosine distance between competency group name and skill embeddings . Similarity from top skills per group : Apart from utilizing the similarity between competency group name and skill , we have also created similarity-based features between a given skill and skills present in the competency group . As an example , we have a skill named auditing and competency group finance . Three similarity-based features were created called top1 , top2 , and top3 , where top1 is cosine similarity score between skill auditing and most similar skill from finance , top2 is the average cosine similarity score of top two most similar skills and top3 is the average cosine similarity score of top three most similar skills . As shown in Appendix Table 5 , the use of similarity-based features beyond three skills did not improve model performance . 2.2.3 CLUSTER-BASED FEATURE . For generating labels for skills using clustering , we experimented with two techniques on SkillBERT embedding – K-means and spectral clustering . Scikit-learn package of K-means was used to generate 45 cluster labels . The number 45 was decided by using the elbow method , the graph of which is shown in Appendix Figure 8 . Using spectral clustering we generated 35 cluster labels . The details of how we used spectral clustering on SkillBERT embedding to generate cluster-based feature are given in Appendix section A.1 . 2.2.4 SKILL TFIDF FEATURE . TFIDF ( Salton & McGill , 1986 ; Ramos , 1999 ) is widely used in text mining to find rare and important words in a document , and as in our training data a single skill can be part of multiple competency groups , we used the same strategy to find skills that are unique to a competency group by calculating their TFIDF value . However , as group information will not be available for new skills , we will calculate the TFIDF of such skills differently . First , we will find the most similar top 3 existing skills and thereafter , take the average of their TFIDF values . This resultant value will be the TFIDF value for a new skill .

Summary: The manuscript focuses on a trending topic of applying a Bidirectional Encoder Representations from Transformers (BERT)-based prediction model to a new domain. More precisely, it addresses classifying Electronic Recruitment Records (ERRs) with respect to job skills. Its contributions include, but are not limited to, (i) releasing a related de-identified ERR dataset to the public domain, (ii) introducing a BERT-based embedding model, called SkillBERT, to group skills present in this ERR dataset into as competency clusters, and (iii) giving experimental evidence of the obtained modelling gains.