SkillBERT: “Skilling” the BERT to classify skills!
Authors: Amber Nigam
Shikha Tyagi
Kuldeep Tyagi
Arpan Saxena
Venue:  (2021)

Abstract: In the age of digital recruitment, job posts can attract a large number of applications, and screening them manually can become a very tedious task. These recruitment records are stored in the form of tables in our recruitment database (Electronic Recruitment Records, referred to as ERRs). We have released a de-identified ERR dataset to the public domain. We also propose a BERT-based model, SkillBERT, the embeddings of which are used as features for classifying skills present in the ERRs into groups referred to as "competency groups". A competency group is a group of similar skills and it is used as matching criteria (instead of matching on skills) for finding the overlap of skills between the candidates and the jobs. This proxy match takes advantage of the BERT's capability of deriving meaning from the structure of competency groups present in the skill dataset. In our experiments, the SkillBERT, which is trained from scratch on the skills present in job requisitions, is shown to be better performing than the pre-trained BERT and the Word2Vec. We have also explored K-means clustering and spectral clustering on SkillBERT embeddings to generate cluster-based features. Both algorithms provide similar performance benefits. Last, we have experimented with different machine learning algorithms like Random Forest, XGBoost, and a deep learning algorithm Bi-LSTM . We did not observe a significant performance difference among the algorithms, although XGBoost and Bi-LSTM perform slightly better than Random Forest. The features created using SkillBERT are most predictive in the classification task, which demonstrates that the SkillBERT is able to capture information about the skills' ontology from the data. We have made the source code and the trained models of our experiments publicly available.

Under review as a conference paper at ICLR 2021
SKILLBERT: "SKILLING" THE BERT TO CLASSIFY SKILLS!
Anonymous authors Paper under double-blind review
ABSTRACT
In the age of digital recruitment, job posts can attract a large number of applications, and screening them manually can become a very tedious task. These recruitment records are stored in the form of tables in our recruitment database (Electronic Recruitment Records, referred to as ERRs). We have released de-identified ERR dataset to the public domair[] We also propose BERT-based model, SkilIBERT, the embeddings of which are used as features for classifying skills present in the ERRs into groups referred to as "competency groups". A competency group is a group of similar skills and it is used as a matching criteria (instead of matching on skills) for finding the overlap of skills between the candidates and the jobs. This proxy match takes advantage of the BERT's s capability of deriving meaning from the structure of competency groups present in the skill dataset. In our experiments, the SkillBERT, which is trained from scratch on the skills present in job requisitions, is shown be better performing than the pre-trained BERT (Devlin al. 2019) and the Word2Vec (Mikolov et al. 2013). We have also explored K K-means clustering (Loyyd 1982) and spectral clustering (Chung 1997) on SkillBERT embeddings to generate cluster-based features. Both algorithms provide similar performance benefits. Last, we have experimented with different machine learning algorithms
like Random Forest (Breiman 2001). XGBoost (Chen & Guestrin, 2016), and a deep learning algorithm Bi-LSTM (Schuster Paliwall 1997; Hochreiter & Scchiidhuber) 1997). We did not observe significant performance difference among the algorithms, although XGBoost and Bi-LSTM perform slightly better than Random Forest. The features created using SkillBERT are most predictive in the classification task, which demonstrates that the SkillBERT is able to capture information about the skills' ontology from the data. We have made the source code and the trained modelsIlff of our experiments publicly available.
INTRODUCTION
Competency group can be thought of as a group of similar skills required for success in a job. For example, skills such as Apache Hadoop, Apache Pig represent competency in Big Data analysis while HTML, Juvascript are part of Front-end competency. Classification of skills into the right competency groups can help in gauging candidate's job interest and automation of the recruitment process. Recently, several contextual word embedding models have been explored on various domain-specific datasets but no work has been done exploring those models on job-skill specific datasets.
Fields like medical and law have already explored these models in their respective domains. Lee et al. (2019) in their BioBERT model trained the BERT model on large biomedical corpus. They found that without changing the architecture too much across tasks, BioBERT beats BERT and previous state-of-the-art models in several biomedical text mining tasks by large difference. Alsentzer et al. (2019) trained publicly released BERT-Base and BioBBER'T-iinetume discharge summaries. They have shown that embeddings formed are superior to general domain or BioBERT specific embeddings for two well established clinical NER tasks and one medical natural language inference task (i2b2 2010 (Uzuner al. 201T), i2b2 2012 (Sun al. 2013a b)), and
models on clinical notes and
MedNLI (Romanov & Shivade 2018)
filefiies.. 1 zip?
d1

---

Under review as a conference paper at ICLR 2021
Beltagy al. 2019) in their model SciBERT leveraged unsupervised pretraining of BERT based model on a large multi-domain corpus of scientific publications. SciBERT significantly outperformed BERT-Base and achieves better results tasks like sequence tagging, sentence classification, and dependency parsing, even compared to some reported BioBERT results on biomedical tasks.
Similarly, Elwany et al. (2019) in their work has shown the improvement in results fine-tuning the BERT model on legal lomain-speciftt corpora. They concluded that fine-tuning BERT gives the best performance and reduces the need for a more sophisticated architecture and/or features. In this paper, we are proposing multi-label competency group classifier, which primarily leverages: SkillBERT, which uses BERT architecture and is trained on the job-skill data from scratch to generate embeddings for skills. These embeddings are used to create several similarity-basee based features to capture the association between skills and group. We have also engineered features through clustering algorithms like spectral clustering on embeddings to attach cluster labels to skills. All these features along with SkillBERT embeddings are used the final classifier achieve the best possible classification accuracy.
2
METHODOLOGY
As no prior benchmark related to job-skill classification is available, we manually assigned each skill in our dataset to more competency groups with the help of the respective domain experts to create training data. We experimented with three different models: pre-trained BERT, Word2vec, and SkillBERT generate word embeddings. Word2vec and SkillBERT on our skill dataset. We created some similarity-basee and cluster- based features on top of these embeddings. Except for these features, some frequency- generated. A detailed explanation of all the steps is mentioned in the next sections. The details of dataset design and feature engineering used for model creation are given in the next sections.
trained from scratch
and group-basee features were also
TRAINING DATA CREATION
Our approach uses a multi-label classification model to predict competency groups for skill. However, prior competency group tagging was available for existing skills, had to manually assign labels for training data creation. For this task, the skill dataset is taken from organization's database which contains 700,000 job requisitions and 2,997 unique skills. The competency groups were created in consultation with domain experts across all major sec- tors. Currently, there exists 40 competency groups in our data representing all major industries. Also within a competency group, we have classified a skill as core or fringe. For example, in marketing competency group, digital marketing is a core skill while creativity Once training data is created, our job is to classify new skill into these 40 competency groups Some skills can belong to more than one category also. For such cases, a skill will have representation in multiple groups. Figure [ shows an overview of the datasets used in this step.
a fringe skill.
2.2 FEATURE ENGINEERING
For feature creation. have experimented with Word2vec and BERT to generate skill embeddings. By leveraging these skill embeddings we created similarity- based features as well. We also used clustering on generated embeddings create cluster-based features. As multiple clustering algorithms are available the literature, we evaluated the most popular clustering algorithms - K-means (Lloyd] 1982) and spectral clustering for experimentation. We have done extensive feature engineering to capture information at skill level, group level, and skill-group combination level. The details of features designed for experiments are given below.
2.2.1 EMBEDDING FEATURES
Traditionally, n-gram based algorithms were used to extract information from text. However, these methods completely ignore the context surrounding word. Hence, have experimented with Word2vec and BERT based architecture to learn embeddings of skills present in training data. The details of how have leveraged them in problem domain given below.
2

---

Under review as a conference paper at ICLR 2021
Requisition (700,000)
Unique (2997)
Training Data
Competency Group (40)
Figure 1: Dataset used for training data creation
Word2vee is one of the most popular techniques to learn word embeddings using a neural network. It uses shallow, two-layer neural network generate n-dimensional embedding for words. To use the Word2vec model on requisition data, we extracted skills from job requisitions and constructed single document. Each sentence of this document represents the skills present in one requisition. As requisition can have multiple skills, we created 2-dimensional list, where the outer dimension specifies the sentence and inner dimension corresponds the skills in that sentence. E.g. if there are two job requisitions called reql and req2 and their corresponding skills are "Java, J2EE" and "Logistic regression, Data visualization, NLP" then outer index 0 corresponds to req and outer index 1 corresponds to req2. Index 0,0 will refer to Java and Index 0.1 will refer to J2EE and so Also before feeding this data for training lowercasing of words, stop word removal and stemming was performed as part of preprocessing. A total of more than 700,000 requisitions were used for model training. We have used embeddings of size 30 which was decided after evaluating model performance on different embedding sizes.
BERT Bidirectional Encoder Representations from Transformers, is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers. Pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-ar models for tasks such as question answering. next sentence prediction, etc. Similar to Word2vec, BERT can also be used to extract fixed-length embeddings of words and sentences, which can further be used features for downstream tasks like classification. But unlike fixed embedding produced by Word2vec, BERT will generate different embedding for an input word based on its left and right context. BERT has shown performance improvement for many natural language processing tasks. However, it has been minimally explored on the job-skill database. Hence, we leveraged BERT architecture on skill data to train the SkillBERT model. We have leveraged ml.p2.xlarge type 12 GIB GPU memory IxK80 GPU available the AWS cloud for SkillBERT training and it took us around 72 hours to completely train it on our dataset. In the next section, we have given the details of training BERT skill corpus.
Training: For training BERT, we used the same corpus as used for Word2vec training and experi- mented with hyperparameters like learning rate and maximum sequence length. For the learning rate, we used 0, 0.05, and 0.01 and finalised 0.01. For maximum sequence length, we used 64, 128, 180 and finalised 128. We could not perform extensive hyperparameter tuning due hardware limitations. Once the training is finished, we extract the last hidden layer output of size 768 and further reduce the embedding size to 128 to decrease the training time of our final model discussed in the experiment section. For the dimensionality reduction of embedding, we did experiments with embeddings of size 32, 64, 128 and 256. As shown in Appendix Table the best results were obtained using embedding of size 128. To make sure information from all the 768 dimensions is leveraged, we trained a 2 2-layer neural network classifier using SkillBERT embeddings as an independent feature and competency group as dependent variable. Out of the 2,997 skills, 80% were used for training and the rest of the 20% were used for the validation. This model generates the probability values of a skill belonging to each of the 40 competency groups and used feature in the final model at skill and competency group combination level. We have referred this feature as "bert-prob" in the rest of the sections. Figure repressens the architecture of the model used for getting these probabilities.
3

---

Under review as a conference paper at ICLR 2021
SKIBBRTT embedding ingut size 768
Hiddon layer 512 nodes
Hidden layer of 125 nodes
layer corresponds to compotency group
Figure Classifier architecture
2.2.2 SIMILARITY-BASER FEATURES
By leveraging skill embeddings generated using embedding techniques, similarity- based features were created to capture the association between group and skill. The details of those given below.
Similarity from competency group: In competency group data, the name of each competency group is also present as a skill. We created a similarity score feature measuring the cosine distance between competency group name and skill embeddings.
Similarity from top skills per group: Apart from utilizing the similarity between competency group name and skill, have also created similarity-base -based features between given skill and skills present in the competency group. As an example, we have a skill named auditing and competency group finance. Three similarity-basee features were created called topl, top2, and top3, where topl cosine similarity score between skill auditing and most similar skill from finance, top2 is the average cosine similarity score of top two most similar skills and top3 is the average cosine similarity score of top three most similar skills. As shown in Appendix Table 5 the use of similarity-based features beyond three skills did not improve model performance.
2.2.3 CLUSTER- BAASED FEATURE
For generating labels for skills using clustering, we experimented with two techniques on SkillBERT embedding K-means and spectral clustering. Scikit-lean package of K-means was used generate 45 cluster labels. The number 45 was decided by using the elbow method, the graph of which is shown in Appendix Figure Using spectral clustering we generated 35 cluster labels. The details of how used spectral clustering SkilIBERT embedding to generate duster-based feature given in Appendix section A.I
2.2.4 SKILL TFIDF FEATURE
TFIDF (Salton & McGill 1986 Ramos 1999) is widely used in text mining to find rare and important words a document, and as in our training data a single skill can be part of multiple competency groups, used the same strategy to find skills that unique to competency group by calculating their TFIDF value. However, as group information will not be available for new skills, we will calculate the TFIDF of such skills differently. First, will find the most similar top 3 existing skills and thereafter, take the average of their TFIDF values. This resultant value will be the TFIDF value for new skill.
2.2.5 CORE AND FRINGE SKILLS
Apart from the features mentioned in the above sections, we have also created group-based features by counting the number of core and fringe skills in each group.
4

---

Under review as a conference paper at ICLR 2021
aisancorit y
(M call back
rig thd Bell
Raade So videemire o ma
Bo
accce control o
aer teee sertee tote ooooeee fae
of beir
sher cnlee senveg
coreidtttanng tanning Will dosug oha chat g
cey
best oifee
rerate aaper
production sesiig
power www.gyyyyyyyyyoooo. chyt coodiaatdon matatreee poren S
Beerg
coficcttig
Sent reidlitth anri dreater
casteme relergee
4g
roctin
domerts
org
costaiic fie
caarg
(a) t-SNE plot skills using pre-trained BERT
(b) t-SNE plot of skills using SkillBERT
Figure 3: t-SNE plot of embeddings of "Customer Support" and "Electronics" competency group. The left image shows the projection generated using pre-trained BERT embedding and the right image is the SkillBERT plot. The top cluster shown in SkillBERT t-SNE plot represents "Electronics" competency group while the bottom cluster represents "Customer Support".
SPS
selvorer
Fright
gita troghngs
groott
2008
2008
gatret
gront inport Escaneettton
OOT Phish
gnteraatonaa jess upstes
gots
retous
9o/ aetnoring
rouing twiccing
contiaation
ginted commur
ptaniig
(a) L-SNE plot of skills using Word2vec
(b) t-SNE plot of skills using SkillBERT
Figure 4: t-SNE plot of embeddings of "Logistic" and "Network" competency group. The left image shows the projection generated using Word2vec embedding and the right image is the SkillBERT plot. The top cluster shown in SkillBERT t-SNE plot represents "Logistic" competency group while the bottom cluster represents "Network".
5

---

Under review as a conference paper at ICLR 2021
Table 1: Machine Learning model Hyperparameters
Machine Learning Models
Best Hyperparameters
Hyperperameter bound N. estimatorss:00-100000 100000 Depth:3-7 N. estimators:400-11 1000. Depth:3-7 400000000
XGBoost Random forest
N essimmators:000 Depth:5
estimators::000, Depth:4
Bi-LSTM
Layers:2(Nodes: 128, 64),
Layers:2 4, Nodes: 32 512,
Optimizer: Dropout:0.2
Dropout: 0.1 -0.5
3 EXPERIMENTS
We have approached the categorization of skills into multiple competency groups multi-label classification problem. We have created our training data at skill X competency group level i.c. for each skill we will have 40 rows, corresponding each competency group. For each skill-group pair, we have tried to predict the probability of that skill belonging to that group using classifier models like XGBoost, Random Forest, and Bi-LSTM. Pairs of models which were compared and had a statistically significant difference in the performance are highlighted with a star in Table 3 The details of all these experiments are given below.
As the first experiment, we did a comparative study among SkilIBERT, pre-trained BERT, and Word2vec models. For pre-trained BERT, we used the bbertbbaseuuncased model which also produces embeddings of size 768. Similar to SkillBERT, we reduced embedding size 128 and generated "bert-prob" feature. All features except cluster labels discussed in the feature engineering section were created using these embedding models. To better analyze the quality of embeddings, we projected high dimensional embeddings of skills present in competency groups in 2-D using t-SNE (van der Maaten & Hinton, 2008). From visualization shown in Figure 3 and Figure 4 it is clear that SkillBERT embeddings reduced the overlapping gap between groups and gave well-defined cluster boundaries compared to word2vec and pre- trained BERT. As a classifier, we used XGBoost and performed hyperparameter tuning through grid-search get the best possible result without over-fitting In the training dataset, there total of 95,904 records and 2,398 unique skills while the testing dataset had 23,976 records and 599 unique skills. The results of this experiment given in Table
SkilBERT vs Word2vec vs Pre-trained BERT:
Table 2: Machine Learning model training time
Machine Learning Models
Training Time (in seconds)
XGBoost
122
Random forest
87
Bi-LSTM
167
K-means vs spectral clustering: In this experiment, we tried to see the effect of adding cluster- based features generated using K-means and spectral clustering on SkillBERT embedding. For this comparison, we applied XGBoost on the cluster labels and the features used in the previous experiment where we compared different embedding approaches.
Random Forest vs Bi-LSTM vs XGBoost: As part of this experiment, we applied Bi-LSTM, Random Forest, XGBoost and spectral clustering based features on SkilIBERT and compared their performance. Table I contains the best performing hyperparameter values and their variation range during tuning through grid-search for all the classifiers used, The number of hyperparameter search trials done was 20, 20, 36 for XGBOOST, Random Forest, and Bi-LSTM models respectively. Table contains the training time each classifier model.
Core fringe skill classification: Finally, we also trained multi-label and multi-class classifier to see how accurately we can classify core and fringe skills. For this, we trained model with 3 classes where, class 0 no label, class 1 fringe skill, and 2 core skill. All the features used in the last experiment were leveraged for this experiment and Bi-LSTM was used as a classifier.
Impact evaluation: By matching competency groups instead of skills, we are broadening the spectrum for matching the skills between candidates and jobs. For instance, candidate has mentioned skills like AngulardS', 'CSS', and 'JavaScript', then there is high probability that the

---

Under review as a conference paper at ICLR 2021
candidate knows 'HMML' too because all these skills belong to "Web development' competency group. Matching using competency groups takes care of similar cases by providing an aggregation to the skills typically required for matching the skills between candidates and jobs. While screening the candidate resumes hiring managers come across many skills which are unknown to them. By normalizing the skills to the competency groups using SkilIBERT, we are reducing the time taken by the hiring managers to find the domain of the skills. The difference in time is because the SkilIBERT not only matches the skills their domains (groups) but also shows constituent skills in each group, thereby providing more context about the groups. As of now, there is no automated way of tracking the resumne screening rate on our platform. However, post introduction of SkillBERT, 150% increase has been observed in the number of average resumes screened per day. The above metric does not account for the confounders like hiring manager's experience and performance among other covariates.
Table 3: Evaluation of result on different embedding models and feature set
Model
Precision
Recall
F1-score
Class 0
Class
Class 0
Class 1
Class 0
Class
*XGBoost + pre-trained BERT
98.83%
51.54%
95.85%
74.26%
97.21% %
60.84%
*XGBoost + Word2vec
98.06%
68.34%
97.36%
65.21%
96.53%
66.73%
XGBoost SkillBERT
99.32%
96.65%
99.47%
84.82%
99.39%
90.35%
XGBoost SkillBERT
K-means
99.27%
96.92%
99.54%
85.24%
99.40%
90.70%
Random forest + SkillBERT + spectral clustering
99.28%
95.15
99.50%
83.48%
99.39%
88.93%
XGBoost SkillBERT
spectral clustering
99.35%
97.23%
99.48%
85.09%
99.41 %
90.76%
*Bi-LSTM SkillBERT
+ spectral clustering
99.26%
95.86%
99.57%
86.43%
99.42%
90.90%
4 RESULTS
Results shown in Table concllude that SkillBERT improved the performance of the classification model over Word2vec and pre-trained BERT. Use of XGBoost with SkillBERT based features give Fl-score of 90.35% for class as compared to 60.83% and 66.73% of pre-trained BERT and Word2vec based features. Use of different machine learning (XGBoost and Random Forest), deep learning (Bi-LSTM) algorithms, and flustering-base features (K-means and spectral clustering) on top of SkillBERT is not making a statistically sigmificant difference and the results are very similar. The difference between the validation dataset and test dataset Fl scores was less than 0.65 and 0.5 percentage points and the variance of validation data F1 scores for different hyperparameter trials was 1.20 and 1.05 percentage points for XGBBost--KKIIBBERTT------ LSTM-SKIIBBERT
clustering and Bi- clustering models respectively. We computed feature importance using the XGBoost model and "bert-prob" explained in section I created using SkillBERT was the top feature in the list. TFIDF and similarity- based features were also highly predictive. Next, the results of experiment 4 (core vs fringe skill classification) given in Table show that were able to classify fringe skills for a group more accurately compared to core skill. All the reported results are statistically significant p<0.05.
7

---

Under review as a conference paper at ICLR 2021
Table 4: Core fringe skill classifier results
Precision
Recall Class
F1-score
Class 0
Class 1
Class 2
Class 0
Class 2
Class 0
Class 1
Class 2
99.07%
93.19%
99.76%
99.74%
78.28%
62.45%
99, 99.40%
85.08%
76.81 %
5 CONCLUSION
In this paper, we have addressed the problem of recruiters manually going through thousands of applications find suitable applicant for the posted job. To reduce the manual intervention, multi-label skill classification model is developed which can classify skills into competency groups and hence, helps in quick mapping of relevant applications a job. For skill representation, different word embedding models like Word2vec and BERT are used and comparison among classification results of different machine learning models is shown. Additionally, features like TFIDE, clustering labels, and similarity-based features are explored for better classification of skills. We trained BERT on a domain- -specffc dataset and a significant improvement is noticed while comparing the results with pre- pre-trained BERT and Word2vee.
REFERENCES
Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd
Clinical Natural Language Processing Workshop, pp. 72-78, Minneapolis, Minnesota, USA,
June 2019, Association for Computational Linguistics. doi: 10.18653/1//119-9999 URL
https: www acl www.:w.wwwwb..o..c www.aclwwb.orgg/atnoooooo www..oooogggooogggoooooooooooo/o 9-1909
Iz. Beltagy, Kyle Lo, and Arman Cohan. SeiBERT: pretrained language model for scientific text.
In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing /EMNLP-IJCNLE pp. 3615-3620, Hong Kong, China, November 2019. Association for Computational Linguistics doi: 10.18653/v1/D199 1371.
Leo Breiman. Random forests. Machine Learning, 45( (5(1):::11))) 2001. ISSN 0885-6125. doi: 10. 1023/A: 10233A:1000933444224 URL http:////00/////........../000.0
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16 pp. 785-794, New York, NY, USA, 2016. ACM. ISBN 978 --52---2----- doi: 10.1 00.14///939772.2937777
1tt..o/......1//939535722223 acm. www.orgg/11./ 11151/0.11122200 0e..44///399722
F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp-
Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653//1/N11--4222 URL https
www.. www.aalwwb. www.:eww..gganaaaaeeoeeee www.e/borrg.aatooooo//eeee
1423
Emad Elwany, Dave Moore, and Gaurav Oberoi. (BERT) goes law school: Quantifying the competitive advantage of access to large legal corpora in contract understanding. In Workshop on Document Intelligence at NeurIPS 2019, 2019. URL htt.:tttt////ot htttps://operrvieerr net forum?
iddrrkeMT 9cLH
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: pre-trained biomedical language representation model for biomedical text
8

---

Under review as a conference paper at ICLR 2021
mining. Bioinformatics, 36(4) 09 2019. ISSN 1367-4803 doi: 10.1099//ioonfommatiss/// btz682.
S. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2):
129-137, 1982.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URLhttp: aariiviivvvv.//.ii org/ abs/130s..... 3781
Juan Ramos. Using tf-idf to determine word relevance document queries, 1999.
Alexey Romanov and Chaitanya Shivade. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1586-1596, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10. 10.18653/11/D18- 11877
URL http:://wwwwwwwww..cooom
anthholgyy/DB81187
Gerard Salton and Michael McGill. Introduction modern information retrieval. 1986.
Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(1 (6:6732273-681,, 1997.
Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. Annotating temporal information in clinical narratives. Journal of biomedical informatics, 46, 07 2013a. doi: 10. 10116ffbbi.2113.07.000
Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. Evaluating temporal relations in clinical text: 2012 i2h2 challenge. Journal of the American Medical Informatics Association JAMIA, 20, 04 2013b. doi: 10.1 3056//mmajnn22011-000000
Özlem Uzuner, Brett R South, Shuying Shen, and Scott L Vall 2010 i262/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 1855) 556, 06 2011. ISSN 1067-5027. doi: 10.1 1336mmmajn 100001110000 1-000203. URL
https:///on .ow.roorroooo.
-nl-nn11--1111 -00110000002
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma-
chine Learning Research, 9:2579-2600 2008. URL BRLt::/://ww/ jmlr r org/ htgppapers/v//
wandermaaten htm
A APPENDIX
A.1 SPECTRAL CLUSTERING
Spectral clustering is widely used unsupervised learning method for clustering. In spectral clustering, the data points are treated as nodes of a graph and these nodes are then mapped to a low-dimensional space using eigenvectors of graph laplacian that be easily segregated to form clusters. Spectral clustering utilizes three matrices, details of those are given below.
1. Similarity graph (Affinity matrix): A similarity graph is pair G (V, A), where V=(u, Um a set of nodes or vertices. Different skills are forming different nodes as shown in Figure A is symmetric matrix called the affinity matrix, such that ba.; 2 0 for all i.j I,..,...,m],, and ba = 0 for i= I......m. We say that a set(v,,), v,,2; is edge if bo O. Where bo.; bert affinity between nodes i and j computed using cosine similarity between SkilIBERT embeddings of the corresponding skills. The corresponding (undirected) graph (V,E) with ( ((,,e,) bo.j >0), is called the underlying graph of G. An example of similarity graph structure affinity matrix is shown in Figure 5
Degree matrix(D): A is an mxm symmetric matrix with zero diagonal entries and with the other entries ba.j R arbitrary, for any node v,EV, the degree of es is defined as
d= d=
Σ bau
(1)

---

Under review as a conference paper at ICLR 2021
Dijial matketng
beren (2)
Palkeing
leping
Oigiul
D0
ba
Dypai
Media
ba
Pubratinng
ba
ba
0
ba
Advetsing (5)
srasding
DR
0
ba
Adectisigg
ba
Figure 5: Adjacency matrix representation of Graph
d = ba ba ba ba
marketing
buylng
Publishing
Branding
Advertsing
d = ba + bag + baz
Dighal
d_
-ba
-ba
-ba
-ba,
dg = ba baya" bag
Media buying
-ba
12
d
-ba
-ba
Publishhing
-ba
-ba
dg
.ba
d. = ba + baz ba.
Branding
-ba,
-ba,,
d,
-ba
d = ba + bags + ba
Advorissin
-ba
-ba
-ba
Figure 6: Graph laplacian for example in Figure
and degree matrix D as
= (d(()) .......... d(v.))
3. Graph laplacian (L): D is a diagonal matrix and A is affinity matrix then we can compute L as follows:-
The Laplacian's 's diagonal is the degree of our nodes, and the off-diagonal is the negative edge weights (similarity between nodes). For clustering the data in more than two clusters, have to modify laplacian to normalize it.
Lnorm D 1-1 1/2
(4)
We know that
Lnorm X = XX
(5)
Where X is the eigenvector of Lnorm corresponding to eigenvalue X. Graph Laplacian is a semi- positive definite matrix and therefore, eigenvalues (A1 X2 ..... An where 0 X 2 > 2 An and eigenvectors X2.X2....X An exam- ple a sample laplacian matrix is given in Figure 6 Once we calculate the eigenvalues of Lnorm and eigenvectors corresponding to smallest eigenvalues where k is number of clusters, create matrix of these eigenvectors stacking them vertically so that every node is represented by the corresponding node representations into k clusters.
eigenvalues are greater than or equals to O. Thus, we get
row of this matrix and use K-means clustering to cluster these For our experiment, we chose the first 35 eigenvectors to create 35 clusters and used them as features for model training. The number 35 was decided using the criteria of difference between consecu- tive eigenvalues. As shown in Figure 7 the difference between eigenvalue 35 and 36 is significantly bigger.
A.2 MISCELLANEOUS
This section contains the results of experiments done for hyperparameter selection and some figures referenced the main text. Figure Shows the elbow method graph for deciding the number of clusters in K-means. 5able(5sshows the results of experiments done for varied number of top skills for similarity based features. Table shows the effect of different SkillBERT embedding size on the results of XGBoost classifier.
10

---

Under review as a conference paper at ICLR 2021
Table 5: Result for different Number of top skills similarity values in feature set (In this experiment, all the features mentioned the experiment section "SkilIBERT vs Word2vec vs Pre-trained BERT" used and only the number of skills used for similarity value calculation were varied. As classifier we used XGBoost)
No. of skills used
Precision
Recall
Fscor
Class 0
Class 1
Class 0
Class
Class
Class
Top skill
99.22%
95.15%
98.89%
83.92%
99.05%
89.18%
Top 2 skills
99.27%
96.10%
99.26%
84.10%
99.26%
89.70%
Top skills
99.32%
96.65%
99.47%
84.82%
99.39%
90.35%
Top skills
99.21%
95.56%
99.40%
84.69%
99.30%
89.80%
0.7
0.6
0.5
Eigenvalue
0.4
0.3
0.2
0.1
0.0
5
10
15
20
35
40
Figure 7: Scatter plot of eigenvalues determine number of eigenvectors and clusters in spectral clustering
Table 6:
Result for different embedding size (In this experiment, XGBoost was used classifier and bert-prob was used along with emdeddings of different sizes as independent variable. No other feature apart from these used)
SkilIBERT embedding size
Precision
Recall
F1-score
Class
Class 1
Class 0
Class
Class
Class
32
98.12%
91.65%
95,47%
80.12%
96.78%
85.50%
64
98.32%
91. 11.8
97. 97.26%
81.10%
97.79%
86. 12%
128
99.12%
92.65%
97.47%
83.80%
98.29%
88.00%
256
99.12%
92.56%
97.40%
83.79%
98.25%
87.96%
11

---

Under review as a conference paper at ICLR 2021
Distortion Cluster Size
350000
300000
200000
150000
50000
0
Figure 8: Elbow method graph to determine the number of clusters in K-means clustering
Table 7: Examples of some candidate and job profiles
Candidate or Job
Skill set
Probable competency groups
Candidatel
Design, knockoutjs, corel draw
Tool design, mechanical design, front end, web development Statistics, production operations
Candidate2
Statistical modeling, statistical
process control
Jobl
Analytical skills, project execution, accounting
Financial operations, business analytics, statistics, accounts Taxation, banking, statistics
Job2
Digital marketing, cash management, ms office, ms excel, ms word, tally
A.3 SKILLBERT TRAINING
The dataset used for training the SkillBERT model can downloaded from here contains the list of skills present in job requisitions. containss examples of some candidate and job profiles. We leveraged Bert-Base architecture on the job-skill data to generate embeddings of size 768, details be found here, Finally, the emheddings generated using the SkillBERT model can be downloaded here
A.4 ANNOTATED DATA PREPARATION
We had domain experts related each field for manual annotation of competency group to a skill for generating the training dataset. Following instructions were given to them:
1. A single skill can belong to multiple groups
If they don't know about a particular skill then they can google and based on that information annotate that skill
3. a skill related to a particular group then they have to further classify it as a core(strongly related) fringe(weakly related) skill to that particular group
The mapping of competency groups and skills can be downloaded here
12

---

Under review as a conference paper at ICLR 2021
Table 8: Feature Description
Feature Name
Feature Type
Dimensionality
bert 0 berr__12 127
SkillBERT Embedding
128
bert-prob
SkillBERT Embedding
1
0-34
Spectral clustering label
35
Valuel-value3
skill-skill similarity
3
tf-idf
TFIDF
1
bert grrp_rrp____ grp_sim
skill-group similarity
1
fore_skill__e sow.illlooutttfriigeeeeeeeeeeeee _aslinngegglgl___l_ ccuunt
group based feature
2
Key
Feature
Dependent
Skill Name
SKIllBERT Embedding
Competency Group (40)
finance
0.31
0.1
10.
autocad
0.21
0
Training (2398)
sqi
0.1
0.99
00
marketing
0.9
0
Validatioo
(599)
0
Figure 9: Data format used for creating bert_ prob feature
A.5 FEATURES
The details of features used in the training of Bi-LSTM model, which gave the best performance are given in Table g
A.6 TRAINING DATA FORMAT
The data format used for generating bert_ prob feature and final model shown in Figure and Figure respectiiv..
A.7 RUNNING THE EXPERIMENT
The code to run all the experiments mentioned in the paper can be downloaded here This codebase uses python 3.6 and all the packages used for this experiment can be downloaded by installing requirements.tx An overview of all the folders present in the code is given below:
This folder contains the main python files used for running the experiments mentioned in the paper. Inside the main() method there are functions for data preparation, training, and testing. We have provided comments in each section for better understating of the modules. The code present in the file "skillbert, setererr__sperar__eeeeeee cirrallcccssstttnnnp_ SkilIBERT and spectral clustering related features which gave us the best performance. You can directly jump to this code if you don't want
training_cone codes:
is used to train Bi-LSTM model on
other intermediary experiments. The experiment for classifying skill into core and fringe can be run using 3 3_class_ced catasssassssss........
Apart from these if you want to other experiments mentioned in the paper, you do by running "word2vec only.py" for classifying skills using only Word2vec model, "skillbert.py" for classifying skills using only SkillBERT model, "berr_pretrain_oo" ooniiiy_nolyp"" for classifying skills using
13

---

Under review as a conference paper at ICLR 2021
Key
Feature
Dependent
Name(2997)
competency
Skill
SkilBERT Embedding (128)
bert_proo
spectral cluster index
bert_srp_st
skill similarity33)
fringe_skil skil count
core_skill
pore_sill_coumt countt
Group(40)
ppc, digital
marketing
0.97
0,97,08550.8
20
Training datal (0X)
0.31 .0.1 Finence
0.63,0.5,0.4
15
0.11 .0.2
0.2
0.89
0.8,0082,066
Val delietion data (-20M)
htmi, web development
0.9
0.2
0.75
0.91,075,008
nio machine learning
0.22 0.1
0.8
0.7
0.78
0.200.3,777
25
Figure 10: Data format used for final model creation
only pre-trained BERT model and "skillbert and kmeans.py" for classifying skills using SkillBERT and k-means on SKillBERT embedding.
2. feature ccautton: This folder contains the code for creating features used for training the models. If you don't want to go through each code, features created using these code files are already available in the feature data folder. Codes present in the training the model training.
_ode code also uses these CSV files directly for
feature data: As mentioned before, this folder contains CSV files of features generated using codes present in feature_creation folder.
4. model: This folder contains the final model trained using all the experiments mentioned in the paper. Folder "skill bert Btrrbbeerrrreeeeeeeeeee__e_eeeee crrrrtspperalllaeeee chtertriic contains the Bi-LSTM model which has been used as the final model.
5. dataset: This folder contains the final training and testing data used for each experiment. You can use these files directly test the corresponding model.
14


Reviews:
[{"review_id": "qTBsLiJ3yRb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied the problem of classifying skills into competency groups. The authors proposed SkillBERT, a BERT-based model, to extract the embeddings of skills and use that for the classification task. \n\nStrengths:\n1. The authors presented the details of feature engineering and experiment design. They also conducted extensive and comprehensive experiments which compare a lot of classification/feature engineering methods. It is a very good practical guide for doing related tasks.\n\n2. The authors released the code and dataset, which largely improve the reproducibility of this work.\n\n3. The paper is well written and easy to follow.\n\nWeakness:\n1. This paper studied a very specific problem which might only be interested to a very small group of researchers. It seems more like an industry track paper. \n\n2. If I am correct, the problem size is relatively small. The total number of skills is 2997, and each of them will be classified into one or more competency group out of 40 competency group. Do we really a BERT model for this problem? Maybe the authors could provide some cost-performance tradeoff analysis here.\n\nQuestions for authors:\n1. In Table 3, I only see the precision/recall/F-1 score for class 0 and class 1. Where is the number for class 2? Did I misunderstand this experiment?\n\nOverall comments:\n\nThis is a good paper and I personally support it to be accepted. However, I do think that it seems more like an industry track paper. This is not for me to decide, maybe Meta-reviewer could share some opinions about whether this paper is suitable to ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eA9wdHVsSLL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript focuses on a trending topic of applying a Bidirectional Encoder Representations from Transformers (BERT)-based prediction model to a new domain. More precisely, it addresses classifying Electronic Recruitment Records (ERRs) with respect to job skills. Its contributions include, but are not limited to, (i) releasing a related de-identified ERR dataset to the public domain, (ii) introducing a BERT-based embedding model, called SkillBERT, to group skills present in this ERR dataset into as competency clusters, and (iii) giving experimental evidence of the obtained modelling gains. \n\nHowever, I am not convinced that these experiments are sufficient to support accepting the manuscript for the following five main reasons:\n\nFirst, the compared models and methods are somewhat old and elementary (e.g., word2vec and TFxIDF), thereby failing to capture the current advances. \n\nSecond, in my opinion, the data annotation process calls for clarifications. For example, what was the expertise of the expert annotators, how were research ethics and informed consenting assured in this process involving human annotators as study participants, and do the authors obtain the rights to release the dataset and/or annotated dataset?\n\nThird, the overall experimental setting does not seem to be adequately captured and justified, and I am unable to find a description of the performed statistical significance testing.\n\nFourth, the manuscript demonstrates only a reasonable understanding of related work in applications to skill/competency demand and existing studies in relevant computational methods/models/data (see, e.g., https://www.aclweb.org/anthology/search/?q=%22job%22 for recent relevant papers from the ACL Anthology that are largely missing from the reference list).\n\nFifth, the manuscript should be edited more carefully by clarifying both its contributions and limitations in relation to related work; describing and justifying its methodology and experiments; moving the in-text citations from the abstract to the body text of the later sections; and enhancing the image readability.\n\nIn conclusion, the study is valuable but needs further work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Tq83-QjVz-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model for job application screening. Since there is no job-related dataset available, the authors manually assigned labels to a large job application dataset. A skill set (e.g., Apache Hadoop, Apache Pig, HTML, Javascript) is firstly extracted from the job dataset.  Then a competency group is constructed (e.g., big data, front-end) as the labels. The problem is then formulated as a multi-label classification problem. That is, given a skill (which may belong to multiple competency groups), the model has to predict its competency groups. The authors proposed to use BERT as the main model. Moreover, the authors use additional features like similarity-based and cluster-based features. The experimental results are good. We think it can help recruiters find a suitable applicant.\n\nHowever, this paper is straightforward. Using BERT as a main model for text classification is a well-known technique, and many papers already applied BERTs in other domains like biomedicine and law. So we think the technical contribution of this paper is limited. Furthermore, some parts of this paper are not clearly explained. For example, the authors mentioned that they also use some features like frequency-based and group-based features, but did not find detailed descriptions of these two features.\n\nOne positive side of this paper is that the authors release a publicly available job application dataset.\nEstablishing a dataset is time-consuming, and requires a lot of human efforts. The dataset consists of 700,000 job requisitions, which is large enough. It is good that the authors are willing to share this dataset.\n\nTo sum up, this paper proposes a skill classification model for job screening, which is useful, but we think that the methodology and its technical contribution are not strong enough. It might not be qualified as a regular paper for ICLR.\n\nSkillBERT: \"Skilling\" the BERT to Classify Skills", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach for the task of categorizing competencies in terms of worker skillsets. This is a potentially useful (if somewhat niche) task, and one strength here is a resource to support further research on the topic. However, the contribution here is limited: The methods considered are not new, and while the problem has some practical importance it does not seem likely to be of particular interest to the broader ICLR community. 
Decision: Reject