SlowGuess/NAACL_2022_OCR huggingface dataset:
SKILLSPAN: Hard and Soft Skill Extraction from English Job Postings
Mike Zhang\* Kristian Norgaard Jensen\* Sif Dam Sonniks Barbara Plank
$\diamond$ Department of Computer Science, IT University of Copenhagen, Denmark  $\clubsuit$ Center for Information and Language Processing (CIS), LMU Munich, Germany
{mikz, krnj, sifs}@itu.dk bplank@cis.uni-muenchen.de
Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowdsourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SKILLSSPAN, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard and soft skills by domain experts. We introduce a BERT baseline (Devlin et al., 2019). To improve upon this baseline, we experiment with language models that are optimized for long spans (Joshi et al., 2020; Beltagy et al., 2020), continuous pre-training on the job posting domain (Han and Eisenstein, 2019; Gururangan et al., 2020), and multi-task learning (Caruana, 1997). Our results show that the domain-adapted models significantly outperform their non-adapted counterparts, and single-task outperforms multi-task learning.
Job markets are under constant development—often due to developments in technology, migration, and digitization—so are the skill sets required. Consequently, job vacancy data is emerging on a variety of platforms in big quantities and can provide insights on labor market skill demands or aid job matching (Balog et al., 2012). SE is to extract the competences necessary from unstructured text.
Previous work in SE shows promising progress, but is halted by a lack of available datasets and annotation guidelines. Two out of 14 studies release their dataset, which limit themselves to crowd-sourced labels (Sayfullina et al., 2018) or annotations from a predefined list of skills on the document-level (Bhola et al., 2020). Additionally,
Figure 1: Examples of Skills & Knowledge Components. Annotated samples of passages in varying job postings. More details are given in Section 4.
none of the 14 previously mentioned studies release their annotation guidelines, which obscures the meaning of a competence. Job markets change, as do the skills in, e.g., the European Skills, Competences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy (Section 3). Hence, it is important to cover for possible emerging skills.
We propose SKILLSPAN, a novel SE dataset annotated at the span-level for skill and knowledge components (SKCs) in job postings (JPs). As illustrated in Figure 1, SKCs can be nested inside skills. SKILLSPAN allows for extracting possibly undiscovered competences and to diminish the lack of coverage of predefined skill inventories.
Our analysis (Figure 2) shows that SKCs contain on average longer sequences than typical Named Entity Recognition (NER) tasks. Albeit we additionally study models optimized for long spans (Joshi et al., 2020; Beltagy et al., 2020), some underperform. Overall, we find specialized domain BERT models (Alsentzer et al., 2019; Lee et al., 2020; Gururangan et al., 2020; Nguyen et al., 2020) perform better than their non-adapted counterparts. We explore the benefits of domain-adaptive pretraining on the JP domain (Han and Eisenstein, 2019; Gururangan et al., 2020). Last, given the examples from Figure 1, we formulate the task as both as a sequence labeling and a multi-task learning (MTL) problem, i.e., training on both skill and knowledge components jointly (Caruana, 1997).
Human Language Technologies, pages 4962 - 4984
Annotations	Approach	Size	Skill Type	(Baseline) Model(s)	♂	♀
Kivimäki et al. (2013)	Document-level	Automatic	N/A	Hard	LogEnt., TF-IDF, LSA, LDA	X	X
Zhao et al. (2015)	Sentence-level	Automatic	N/A	Hard	Word2Vec	X	X
Javed et al. (2017)	Span-level	Skill Inventory	N/A	Both	Word2Vec	X	X
Jia et al. (2018)	Span-level	Automatic	21,158 JPs*	Hard	LSTM	X	X
Sayfullina et al. (2018)	Span-level	Crowdsourcing	4,863 Sent.	Soft	CNN, LSTM	✓	X
Smith et al. (2019)	Span-level	Manual	100 JPs	Hard	Pattern Matching	X	X
Gugnani and Misra (2020)	Span-level	Domain Experts	~200 JPs	Hard	Word2Vec, Doc2Vec	X	X
Li et al. (2020)	Document-level	Proprietary	N/A	Hard	FastText	X	X
Shi et al. (2020)	Span-level	Proprietary	N/A	Hard	FastText, USE, BERT	X	X
Tamburri et al. (2020)	Sentence-level	Domain Experts	~3,000 Sent.	Both	BERT	X	X
Chernova (2020)	Span-level	Manual	100 JPs	Both	FinBERT	X	X
Bhola et al. (2020)	Document-level	Skill Inventory	20,298 JPs*	Hard	BERT	✓	X
Smith et al. (2021)	Span-level	Manual	100 JPs	Hard	Pattern Match., Word2Vec	X	X
Liu et al. (2021)	Document-level	Crowdsourcing	N/A	Hard	GNN	X	X
This work	Span-level	Domain Experts	391 JPs	Both	(Domain-adapted) BERT	✓	✓
Table 1: Contributions of Related Work. We list the recent works of Skill Extraction. Note that (*) indicates labels that are automatically inferred from some source (e.g., a predefined skill inventory) and not manually annotated. With respect to the annotation approach, "Manual" indicates uncertainty whether they used domain experts or not. Also note that many works do not release their dataset with annotations (♀) nor guidelines (♂). The list is inspired by Khaouja et al. (2021).
Contributions In this paper: ① We release SKILLSPAN, a novel skill extraction dataset, with annotation guidelines, and our open-source code. ② We present strong baselines for the task including a new SpanBERT (Joshi et al., 2020) trained from scratch, and domain-adapted variants (Gururangan et al., 2020), which we will release on the HuggingFace platform (Wolf et al., 2020). To the best of our knowledge, we are the first to investigate the extraction of skills and knowledge from job postings with state-of-the-art language models. ③ We give an analysis on single-task versus multi-task learning in the context of skill extraction, and show that for this particular task single-task learning outperforms multi-task learning.
There is a pool of prior work relating to SE. We summarize it in Table 1, depicting state-of-the-art approaches, level of annotations, what kind of competences are annotated, the modeling approaches, the size of the dataset (if available), type of skills annotated for, baseline models, and whether they release their annotations and guidelines.
As can be seen in Table 1, many works do not release their data (apart from Sayfullina et al., 2018 and Bhola et al., 2020) and none release their annotation guidelines. In addition, none of the previous studies approach SE as a span-level extraction task with state-of-the-art language models, nor
did they release a dataset of this magnitude with manually annotated (long) spans of competences by domain experts.
Although Sayfullina et al. (2018) annotated on the span-level (thus being useful for SE) and release their data, they instead explored several approaches to Skill Classification. To create the data, they extracted all text snippets containing one soft skill from a predetermined list. Crowdworkers then annotated the highlighted skill whether it was a soft skill referring to the candidate or not. They show that an LSTM (Hochreiter et al., 1997) performs best on classifying the skill in the sentence. In our work, we annotated a dataset three times their size (Table 2) for both hard and soft skills. In addition, we also extract the specific skills from the sentence.
Tamburri et al. (2020) classifies sentences that contain skills in the JP. The authors manually labeled their dataset with domain experts. They annotated whether a sentence contains a skill or not. Once the sentence is identified as containing a skill, the skill cited within is extracted. In contrast, we directly annotate for the span within the sentence.
Bhola et al. (2020) cast the task of skill extraction as a multi-label skill classification at the document-level. There is a predefined set of unique skills given the job descriptions and they predict multiple skills that are connected to a given job description using BERT (Devlin et al., 2019). In addition, they experiment with several additional layers for better prediction performance. We instead explore domain-adaptive pre-training for SE.
'https://github.com/kris927b/SkillSpan
The work closest to ours is by Chernova (2020), who approach the task similarly with span-level annotations (including longer spans) but approach this for the Finnish language. It is unclear whether they annotated by domain experts. Also, neither the data nor the annotation guidelines are released. For a comprehensive overview with respect to SE, we refer to Khaouja et al. (2021).
3 Skill & Knowledge Definition
There is an abundance of competences and there have been large efforts to categorize them. For example, the The International Standard Classification of Occupations (ISCO; Elias, 1997) is one of the main international classifications of occupations and skills. It belongs to the international family of economic and social classifications. Another example, the European Skills, Competences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy is the European standard terminology linking skills and competences and qualifications to occupations and derived from ISCO. The ESCO taxonomy mentions three categories of competences: Skill, knowledge, and attitudes. ESCO defines knowledge as follows:
"Knowledge means the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study." 2
For example, a person can acquire the Python programming language through learning. This is denoted as a knowledge component and can be considered a hard skill. However, one also needs to be able to apply the knowledge component to a certain task. This is known as a skill component. ESCO formulates it as:
"Skill means the ability to apply knowledge and use know-how to complete tasks and solve problems." 3
In ESCO, the soft skills are referred to as attitudes. ESCO considers attitudes as skill components:
"The ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations
$^{2}$ https://ec.europa.eu/esco/portal/escopedia/Knowledge
3https://ec.europa.eu/esco/portal/ scopedia/Skill
↓ Statistics, Src. →	BIG	HOUSE	TECH	Total
Train	# Posts	60	60	80	200
# Sentences	1,036	1,674	3,156	5,866
# Tokens	29,064	36,995	56,549	122,608
# Skill Spans	1,086	984	1,237	3,307
# Knowledge Spans	439	781	2,188	3,408
# Overlapping Spans	45	29	135	209
Development	# Posts	30	30	30	90
# Sentences	783	1,022	2,187	3,992
# Tokens	11,762	19,173	21,149	52,084
# Skill Spans	469	525	545	1,539
# Knowledge Spans	126	287	806	1,219
# Overlapping Spans	12	17	32	61
Test	# Posts	36	33	32	101
# Sentences	1,112	1,216	2,352	4,680
# Tokens	14,720	21,923	20,885	57,528
# Skill Spans	634	637	459	1,730
# Knowledge Spans	242	350	834	1,426
# Overlapping Spans	12	8	9	29
Total	# Posts	126	123	142	391
# Sentences	2,931	3,912	7,695	14,538
# Tokens	55,546	78,091	98,583	232,220
# Skill Spans	2,189	2,146	2,241	6,576
# Knowledge Spans	807	1,418	3,828	6,053
# Overlapping Spans	69	54	178	301
N	# Posts	126,769
# Sentences	3,195,585
# Tokens	460,484,670
Table 2: Statistics of Dataset. Indicated is the number of JPs across splits & source and their respective number of sentences, tokens, and spans. The total is reported in the cyan column and rows. We report the overall statistics of the unlabeled JPs  $(\mathcal{U})$  in the gray rows.
and professional and personal development."4
To sum up, hard skills are usually referred to as knowledge components, and applying these hard skills to something is considered a skill component. Then, soft skills are referred to as attitudes, these are part of skill components. There has been no work, to the best of our knowledge, in annotating skill and knowledge components in JPs.
4 SKILLSPAN Dataset
Data We continuously collected JPs via web data extraction between June 2020-September 2021. Our JPs come from the three sources:
1. BIG: A large job platform with various types of JPs, with several type of positions;
2. HOUSE: A static in-house dataset consisting of similar types of jobs as BIG. Dates range from 2012-2020;
<sup>4</sup>http://data.europa.eu/esco/skill/A  
<sup>5</sup>Our data statement (Bender and Friedman, 2018) can be found in Appendix A.
Figure 2: Violin Plots of Annotated Components. Indicated are the distributions regarding the length of spans in each type of annotated component (i.e., length of skills and knowledge components). The white dot is the median length, the bars range from the first quartile to the third quartile, and the colored line ranges from the lower adjacent value to the higher adjacent value.
3. TECH: The StackOverflow JP platform that consisted mostly of technical jobs (e.g., developer positions).
We release the anonymized raw data and annotations of the parts with permissible licenses, i.e., HOUSE (from a governmental agency which is our collaborator) and TECH. For anonymization, we perform it via manual annotation of job-related sensitive and personal data regarding Organization, Location, Contact, and Name following the work by Jensen et al. (2021). Table 2 shows the statistics of SKILLSpan, with 391 annotated JPs from the three sources containing 14.5K sentences and 232.2K tokens. The unlabeled JPs (only to be released as pre-trained model) consist of 126.8K posts, 3.2M sentences, and 460.5M tokens. What stands out is that there are 2-5 times as many annotated knowledge components in TECH in contrast to the other sources, despite a similar amount of JPs. We expect this to be due the numerous KCs depicted in this domain (e.g., programming languages), while we observe considerably fewer soft skills (e.g., "work flexibly"). The amount of skills is more balanced across the three sources. Furthermore, overlapping spans follow a consistent trend among splits, with the train split containing the most.
Data Annotation We annotate competences related to SKCs in two levels as illustrated in Figure 1. We started the process in March 2021, with initial annotation rounds to construct and refine the annotation guidelines (as outlined further below).
The annotation process spanned eight months in total. Our final annotation guidelines can be found in Appendix B. The guidelines were developed by largely following example spans given in the ESCO taxonomy. However, at this stage, we focus on span identification, and we do not take the fine-grained taxonomy codes from ESCO for labeling the spans, leaving the mapping to ESCO and taxonomy enrichment as future work.
Further Details on the Annotation Process The development of the annotation guidelines and our annotation process is depicted as follows: ① We wrote base guidelines derived from a small number of JPs. ② We had three pre-rounds consisting of three JPs each. After each round, we modified, improved and finalized the guidelines. ③ Then, we had three longer-lasting annotation rounds consisting of 30 JPs each. We re-annotated the previous 11 JPs in ① and ②. ④ After these rounds, one of the annotators (the hired linguist) annotated JPs in batches of 50. The data in ①, ②, and ③ was annotated by three annotators (101 JPs).
We used an open source text annotation tool named DOCCANO (Nakayama et al., 2018). There are around  $57.5\mathrm{K}$  tokens (approximately  $4.6\mathrm{K}$  sentences, in 101 job posts) that we calculated agreement on. The annotations were compared using Cohen's  $\kappa$  (Fleiss and Cohen, 1973) between pairs of annotators, and Fleiss'  $\kappa$  (Fleiss, 1971), which generalises Cohen's  $\kappa$  to more than two concurrent annotations. We consider two levels of  $\kappa$  calculations: TOOL is calculated on the token level, comparing the agreement of annotators on each token (including non-entities) in the annotated dataset. SPAN refers to the agreement between annotators on the exact span match over the surface string, regardless of the type of SKC, i.e., we only check the position of tag without regarding the type of the entity. The observed agreements scores over the three annotators from step ③ are between 0.70–0.75 Fleiss'  $\kappa$  for both levels of calculation which is considered a substantial agreement (Landis and Koch, 1977) and a  $\kappa$  value greater than 0.81 indicates almost perfect agreement. Given the difficulty of this task, we consider the aforementioned  $\kappa$  score to be strong. Particularly, we observed a large improvement in annotation agreement from the earlier rounds (step ① and ②), where our Fleiss'  $\kappa$  was 0.59 on token-level and 0.62 for the span-level.
Overall, we observe higher annotator agreement for knowledge components (3-5% higher) com
6Links to our data can be found at https://github. com/kris927b/SkillSpan.
BIG	HOUSE	TECH
SKILL	ambitious	structured	hands-on
proactive	teaching	communication skills
work independently	communication skills	leadership
attention to detail	project management	passionate
motivated	drive	open-minded
KNOWLEDGE	full uk driving licence	english	java
sap energy assessments	supply chain	javascript
right to work in the uk	project management	aws
sen	powders	docker
acca/aca	machine learning	node.js
Table 3: Most Frequent Skills in the Development Data. Top-5 skill components in our data in terms of frequency on different sources. A larger example can be found in Table 8 and Table 9 (Appendix C).
pared to skills which tend to be longer. The TECH domain is the most consistent for agreement while BIG shows more variation over rounds, likely due to the broader nature of the domains of JPs.
Annotation Span Statistics A challenge of annotating spans is the length (i.e., boundary), SKCs being in different domains (e.g., business versus technical components), and frequently written differently, e.g., "being able to work together" v.s. "teamwork"). Figure 2 shows the statistics of our annotations in violin plots. For the training set, the median length (white dot) of skills is around 4 for BIG and HOUSE, for TECH this is a median of 5. In the development set, the median stays at length 4 across all sources. Another notable statistic is the upper and lower percentile of the length of skills and knowledge, indicated with the thick bars. Here, we highlight the fact that skill components could consist of many tokens, for example, up to length 7 in the HOUSE source split (see blue-colored violins). For knowledge components, the spans are usually shorter, where it is consistently below 5 tokens (see orange-colored violins). All statistics follow a similar distribution across train, development, and sources in terms of length and distribution. This gives a further strong indication that consistent annotation length has been conducted across splits and sources.
Qualitative Analysis of Annotations Qualitative differences in SKCs over the three sources are shown (lowercased) in Table 3. With respect to skill components, all sources follow a similar usage of skills. The annotated skills mostly relate to the attitude of a person and hence mostly consist of soft skills. With respect to knowledge components, we observe differences between the three sources. First, on the source-level, the knowledge
components vastly differ between BIG and TECH. BIG postings seem to cover more business related components, whereas TECH has more engineering components. HOUSE seems to be a mix of the other two sources. Lastly, note that both the skill and knowledge components between the splits diverge in terms of the type of annotated spans, which indicates a variation in the annotated components. We show the top-10 skills annotated in the train, development, and test splits for SKCs in Appendix C. From a syntactic perspective, skills frequently consist of noun phrases, verb phrases, or adjectives (for soft skills). Knowledge components usually consists of nouns or proper nouns, such as "python", "java", and so forth.
The task of SE is formulated as a sequence labeling problem. Formally, we consider a set of JPs  $\mathcal{D}$ , where  $d\in \mathcal{D}$  is a set of sequences (i.e., entire JPs) with the  $i^{\mathrm{th}}$  input sequence  $\mathcal{X}_d^i = \{x_1,x_2,\dots,x_T\}$  and a target sequence of BIO-labels  $\mathcal{Y}_d^i = \{y_1,y_2,\dots,y_T\}$  (e.g., "B-SKILL", "I-KNOWLEDGE", "O"). The goal is to use  $\mathcal{D}$  to train a sequence labeling algorithm  $h:\mathcal{X}\mapsto \mathcal{Y}$  to accurately predict entity spans by assigning an output label  $y_{t}$  to each token  $x_{t}$ .
As baseline we consider BERT and we investigate more recent variants, and we also train models from scratch. Models are chosen due to their state-of-the-art performance, or in particular, for their strong performance on longer spans.
$\mathbf{BERT}_{\mathrm{base}}$  (Devlin et al., 2019) An out-of-the-box  $\mathbf{BERT}_{\mathrm{base}}$  model (bert-base-cased) from the HuggingFace library (Wolf et al., 2020) functioning as a baseline.
SpanBERT (Joshi et al., 2020) A BERT-style model that focuses on span representations as opposed to single token representations. SpanBERT is trained by masking contiguous spans of tokens and optimizing two objectives: (1) masked language modeling, which predicts each masked token from its own vector representation. (2) The span boundary objective, which predicts each masked token from the representations of the unmasked tokens at the start and end of the masked span.
We train a SpanBERT $_{\text{base}}$  model from scratch on the BooksCorpus (Zhu et al., 2015) and English Wikipedia using cased Wordpiece tokens (Wu et al., 2016). We use AdamW (Kingma and Ba, 2015) for
Figure 3: Performance of Models. We test the models on SKILLS, KNOWLEDGE, and COMBINED. We report the span-F1 and standard deviation (error bars) of runs on five random seeds. Note that the y-axis starts from 50 span-F1. STL indicates single-task learning and MTL indicates the multi-task model. Differences can be seen on the test set: JobSpanBERT performs best on SKILLS, JobBERT is best on KNOWLEDGE, and JobBERT achieves best in COMBINED. Exact numbers of the plots are in Table 5 (Appendix E).
2.4M training steps with batches of 256 sequences of length 512. The learning rate is warmed up for 10K steps to a maximum value of  $1e - 4$ , after which it has a decoupled weight decay (Loshchilov and Hutter, 2019) of 0.1. We add a dropout rate of 0.1 across all layers. Pretraining was done on a v3-8 TPU on the GCP and took 14 days to complete. We take the official TensorFlow implementation of SpanBERT by Ram et al. (2021).
JobBERT<sup>7</sup> We apply domain-adaptive pre-training (Gururangan et al., 2020) to a BERT<sub>base</sub> model using the 3.2M unlabeled JP sentences (Table 2). Domain-adaptive pre-training relates to the continued self-supervised pre-training of a large language model on domain-specific text. This approach improves the modeling of text for downstream tasks within the domain. We continue training the BERT model for three epochs (default in HuggingFace) with a batch size of 16.
JobSpanBERT<sup>8</sup> We apply domain-adaptive pretraining to our SpanBERT on 3.2M unlabeled JP sentences. We keep parameters identical to the vanilla SpanBERT, but change the number of steps
to 40K to have three passes over the unlabeled data.
Experiments We have 391 annotated JPs (Table 2) that we divide across three splits: Train, dev. and test set. We use 101 JPs that all three annotators annotated as the gold standard test set with aggregated annotations via majority voting. The 101 postings are divided between the sources as: 36 BIG, 33 HOUSE, and 32 TECH. The remaining 290 JPs were annotated by one annotator. We use 90 JPs (30 from each source, namely BIG, HOUSE, and TECH) as the dev. set. The remaining 200 JPs are used as the train set. The sources in the train set are divided into 60 BIG, 60 HOUSE, and 80 TECH.