Venue: NAACL 2022

# SkillSpan: Hard and Soft Skill Extraction from English Job Postings

Mike Zhang\({}^{*}\)\({}^{\diamond}\) Kristian Norgaard Jensen\({}^{*}\) Sif Dam Sonniks\({}^{\diamond}\) Barbara Plank\({}^{\diamond}\)\({}^{\clubsuit}\)

\({}^{\diamond}\)Department of Computer Science, IT University of Copenhagen, Denmark

\({}^{\spadesuit}\)Center for Information and Language Processing (CIS), LMU Munich, Germany

{mikz, krnj, sifs}@itu.dk bplank@cis.uni-muenchen.de

Equal contribution.

###### Abstract

Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowdsourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SkillSpan, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard _and_ soft skills by domain experts. We introduce a BERT baseline Devlin et al. (2019). To improve upon this baseline, we experiment with language models that are optimized for long spans Joshi et al. (2020); Beltagy et al. (2020), continuous pre-training on the job posting domain Han and Eisenstein (2019); Gururangan et al. (2020), and multi-task learning Caruana (1997). Our results show that the domain-adapted models significantly outperform their non-adapted counterparts, and single-task outperforms multi-task learning.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4962 - 4984

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Job markets are under constant development--often due to developments in technology, migration, and digitization--so are the skill sets required. Consequentially, job vacancy data is emerging on a variety of platforms in big quantities and can provide insights on labor market skill demands or aid job matching Balog et al. (2012). SE is to extract the competences necessary from unstructured text.

Previous work in SE shows promising progress, but is halted by a lack of available datasets and annotation guidelines. Two out of 14 studies release their dataset, which limit themselves to crowd-sourced labels Sayfullina et al. (2018) or annotations from a predefined list of skills on the document-level Bhola et al. (2020). Additionally, none of the 14 previously mentioned studies release their annotation guidelines, which obscures the meaning of a competence. Job markets change, as do the skills in, e.g., the European Skills, Comptences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy (Section3). Hence, it is important to cover for possible emerging skills.

We propose SkillSpan, a novel SE dataset annotated at the span-level for _skill_ and _knowledge_ components (SKCs) in job postings (JPs). As illustrated in Figure1, SKCs can be nested inside skills. SkillSpan allows for extracting possibly undiscovered competences and to diminish the lack of coverage of predefined skill inventories.

Our analysis (Figure2) shows that SKCs contain on average longer sequences than typical Named Entity Recognition (NER) tasks. Albeit we additionally study models optimized for long spans Joshi et al. (2020); Beltagy et al. (2020), some underperform. Overall, we find specialized domain BERT models Alsentzer et al. (2019); Lee et al. (2020); Gururangan et al. (2020); Nguyen et al. (2020) perform better than their non-adapted counterparts. We explore the benefits of domain-adaptive pre-training on the JP domain Han and Eisenstein (2019); Gururangan et al. (2020). Last, given the examples from Figure1, we formulate the task as both as a sequence labeling and a multi-task learning (MTL) problem, i.e., training on both skill and knowledge components jointly Caruana (1997).

[FIGURE:A1.F1][ENDFIGUR

ContributionsIn this paper: 1 We release SkillSpan, a novel skill extraction dataset, with annotation guidelines, and our open-source code.1

Footnote 1: [https://github.com/kris927b/SkillSpan](https://github.com/kris927b/SkillSpan)

2We present strong baselines for the task including a new SpanBERT Joshi et al. (2020) trained from scratch, and domain-adapted variants Gururangan et al. (2020), which we will release on the HuggingFace platform Wolf et al. (2020). To the best of our knowledge, we are the first to investigate the extraction of skills and knowledge from job postings with state-of-the-art language models. 3 We give an analysis on single-task versus multi-task learning in the context of skill extraction, and show that for this particular task single-task learning outperforms multi-task learning.

## 2 Related Work

There is a pool of prior work relating to SE. We summarize it in Table 1, depicting state-of-the-art approaches, level of annotations, what kind of competences are annotated, the modeling approaches, the size of the dataset (if available), type of skills annotated for, baseline models, and whether they release their annotations and guidelines.

As can be seen in Table 1, many works do _not release their data_ (apart from Sayfullina et al., 2018 and Bhola et al., 2020) and **none release their annotation guidelines**. In addition, none of the previous studies approach SE as a span-level extraction task with state-of-the-art language models, nor did they release a dataset of this magnitude with manually annotated (long) spans of competences by domain experts.

Although Sayfullina et al. (2018) annotated on the span-level (thus being useful for SE) and release their data, they instead explored several approaches to _Skill Classification_. To create the data, they extracted all text snippets containing _one_ soft skill from a predetermined list. Crowdworkers then annotated the highlighted skill whether it was a soft skill referring to the candidate or not. They show that an LSTM Hochreiter et al. (1997) performs best on classifying the skill in the sentence. In our work, we annotated a dataset three times their size (Table 2) for both hard _and_ soft skills. In addition, we also extract the specific skills from the sentence.

Tamburri et al. (2020) classifies sentences that contain skills in the JP. The authors manually labeled their dataset with domain experts. They annotated whether a sentence contains a skill or not. Once the sentence is identified as containing a skill, the skill cited within is extracted. In contrast, we directly annotate for the span within the sentence.

Bhola et al. (2020) cast the task of skill extraction as a multi-label skill classification at the document-level. There is a predefined set of unique skills given the job descriptions and they predict multiple skills that are connected to a given job description using BERT Devlin et al. (2019). In addition, they experiment with several additional layers for better prediction performance. We instead explore domain-adaptive pre-training for SE.

\begin{table}
\begin{tabular}{l|p{45.5pt} p{45.5pt} l c c c c} \hline  & **Annotations** & **Approach** & **Size** & **skill Type** & **(Baseline) Model(s)** & ✔** \\ \hline Kivimäki et al. (2013) & Document-level & Automatic & N/A & Hard & LogEnt., TF-IDF, LSA, LDA & ✗ \\ Zhao et al. (2015) & Sentence-level & Automatic & N/A & Hard & Word2Vec & ✗ \\ Javed et al. (2017) & Span-level & Skill Inventory & N/A & Both & Word2Vec & ✗ \\ Jia et al. (2018) & Span-level & Automatic & 21,158 JPs* & Hard & LSTM & ✗ \\ Sayfullina et al. (2018) & Span-level & Crowdsourcing & 4,863 Sent. & Soft & CNN, LSTM & ✓ & ✗ \\ Smith et al. (2019) & Span-level & Manual & 100 JPs & Hard & Pattern Matching & ✗ & ✗ \\ Gugnani and Misra (2020) & Span-level & Domain Experts & \textgreater{}\(\sim\)200 JPs & Hard & Word2Vec, Doc2Vec & ✗ & ✗ \\ Li et al. (2020) & Document-level & Proprietary & N/A & Hard & FastText & ✗ & ✗ \\ Shi et al. (2020) & Span-level & Proprietary & N/A & Hard & FastText, USE, BERT & ✗ & ✗ \\ Tamburri et al. (2020) & Sentence-level & Domain Experts & \textasciramento{}\(\sim\)3,000 Sent. & Both & BERT & ✗ & ✗ \\ Chernova (2020) & Span-level & Manual & 100 JPs & Both & FinBERT & ✗ & ✗ \\ Bhola et al. (2020) & Document-level & Skill Inventory & 20,298 JPS* & Hard & BERT & ✓ & ✗ \\ Smith et al. (2021) & Span-level & Manual & 100 JPs & Hard & Pattern Match., Word2Vec & ✗ & ✗ \\ Liu et al. (2021) & Document-level & Crowdsourcing & N/A & Hard & GNN & ✗ & ✗ \\ \hline
**This work** & Span-level & Domain Experts & 391 JPs & Both & (Domain-adapted) BERT & ✓ & ✓ \\ \hline \end{tabular}
\end{table}
Table 1: **Contributions of Related Work**. We list the recent works of Skill Extraction. Note that (*) indicates labels that are automatically inferred from some source (e.g., a predefined skill inventory) and not _manually_ annotated. With respect to the annotation approach, “Manual” indicates uncertainty whether they used domain experts or not. Also note that many works do not release their dataset with annotations ✔ nor guidelines (2). The list is inspired by Khaouja et al. (2021).

 

The work closest to ours is by Chernova (2020), who approach the task similarly with span-level annotations (including longer spans) but approach this for the Finnish language. It is unclear whether they annotated by domain experts. Also, neither the data nor the annotation guidelines are released. For a comprehensive overview with respect to SE, we refer to Khaouja et al. (2021).

## 3 Skill & Knowledge Definition

There is an abundance of competences and there have been large efforts to categorize them. For example, the The International Standard Classification of Occupations (ISCO; Elias, 1997) is one of the main international classifications of occupations and skills. It belongs to the international family of economic and social classifications. Another example, the European Skills, Comperences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy is the European standard terminology linking skills and competences and qualifications to occupations and derived from ISCO. The ESCO taxonomy mentions three categories of competences: _Skill_, _knowledge_, and _attitudes_. ESCO defines knowledge as follows:

"Knowledge means the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study." 2

Footnote 2: [https://ec.europa.eu/esco/portal/escopedia/Knowledge](https://ec.europa.eu/esco/portal/escopedia/Knowledge)

For example, a person can acquire the Python programming language through learning. This is denoted as a _knowledge_ component and can be considered a _hard skill_. However, one also needs to be able to apply the knowledge component to a certain task. This is known as a _skill_ component. ESCO formulates it as:

"Skill means the ability to apply knowledge and use know-how to complete tasks and solve problems." 3

Footnote 3: [https://ec.europa.eu/esco/portal/escopedia/Skill](https://ec.europa.eu/esco/portal/escopedia/Skill)

In ESCO, the _soft skills_ are referred to as _attitudes_. ESCO considers attitudes as skill components:

"The ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and professional and personal development." 4

To sum up, hard skills are usually referred to as _knowledge_ components, and applying these hard skills to something is considered a _skill_ component. Then, soft skills are referred to as _attitudes_, these are part of skill components. There has been no work, to the best of our knowledge, in annotating skill and knowledge components in JPs.

## 4 SkillSpan Dataset

Data5We continuously collected JPs via web data extraction between June 2020-September 2021. Our JPs come from the three sources:

Footnote 5: Our data statement (Bender and Friedman, 2018) can be found in Appendix A.

1. **Big**: A large job platform with various types of JPs, with several type of positions;
2. **House**: A _static_ in-house dataset consisting of similar types of jobs as Big. Dates range from 2012-2020;

\begin{table}
\begin{tabular}{l|c r r r||r}  & \multicolumn{1}{l}{\(\downarrow\)**Statistics, Src. \(\rightarrow\)**} & \multicolumn{1}{l}{**Big**} & \multicolumn{1}{l||}{**Other**} & \multicolumn{1}{l||}{**Tech**} & \multicolumn{1}{l}{**Total**} \\ \hline  & **\# Posts** & 60 & 60 & 80 & 200 \\  & **\# Sentences** & 1,036 & 1,674 & 3,156 & 5,866 \\  & **\# Tokens** & 29,064 & 36,995 & 56,549 & 122,608 \\  & **\# Skill Spans** & 1,086 & 984 & 1,237 & 3,307 \\  & **\# Knowledge Spans** & 439 & 781 & 2,188 & 3,408 \\  & **\# Overlapping Spans** & 45 & 29 & 135 & 209 \\ \cline{2-6}  & **\# Posts** & 30 & 30 & 30 & 90 \\  & **\# Sentences** & 783 & 1,022 & 2,187 & 3,992 \\  & **\# Tokens** & 11,762 & 19,173 & 21,149 & 52,084 \\  & **\# Skill Spans** & 469 & 525 & 545 & 1,539 \\  & **\# Knowledge Spans** & 126 & 287 & 806 & 1,219 \\  & **\# Overlapping Spans** & 12 & 17 & 32 & 61 \\ \cline{2-6}  & **\# Posts** & 36 & 33 & 32 & 101 \\  & **\# Sentences** & 1,112 & 1,216 & 2,352 & 4,680 \\  & **\# Tokens** & 14,720 & 21,923 & 20,885 & 57,528 \\  & **\# Skill Spans** & 634 & 637 & 459 & 1,730 \\  & **\# Knowledge Spans** & 242 & 350 & 834 & 1,426 \\  & **\# Overlapping Spans** & 12 & 8 & 9 & 29 \\ \hline \hline  & **\# Posts** & 126 & 123 & 142 & 391 \\  & **\# Sentences** & 2,931 & 3,912 & 7,695 & 14,538 \\  & **\# Tokens** & 55,546 & 78,091 & 98,583 & 232,220 \\  & **\# Skill Spans** & 2,189 & 2,146 & 2,241 & 6,576 \\  & **\# Knowledge Spans** & 807 & 1,418 & 3,828 & 6,053 \\  & **\# Overlapping Spans** & 69 & 54 & 178 & 301 \\ \hline \hline  & **\# Posts** & & & & 126,769 \\  & **\# Sentences** & & & & 3,195,585 \\  & **\# Tokens** & & & & 460,484,670 \\ \end{tabular}
\end{table}
Table 2: **Statistics of Dataset. Indicated is the number of JPs across splits & source and their respective number of sentences, tokens, and spans. The total is reported in the cyan column and rows. We report the overall statistics of the unlabeled JPs (\(\mathcal{U}\)) in the gray rows.

 

3. **Tech**: The StackOverflow JP platform that consisted mostly of technical jobs (e.g., developer positions).

We release the anonymized raw data and annotations of the parts with permissible licenses, i.e., House (from a governmental agency which is our collaborator) and Tech.6 For anonymization, we perform it via manual annotation of job-related sensitive and personal data regarding Organization, Location, Contact, and Name following the work by Jensen et al. (2021). Table 2 shows the statistics of SkillSpan, with 391 annotated JPs from the three sources containing 14.5K sentences and 232.2K tokens. The unlabeled JPs (only to be released as pre-trained model) consist of 126.8K posts, 3.2M sentences, and 460.5M tokens. What stands out is that there are 2-5 times as many annotated knowledge components in Tech in contrast to the other sources, despite a similar amount of JPs. We expect this to be due the numerous KCs depicted in this domain (e.g., programming languages), while we observe considerably fewer soft skills (e.g., "work flexibly"). The amount of skills is more balanced across the three sources. Furthermore, overlapping spans follow a consistent trend among splits, with the train split containing the most.

Footnote 6: Links to our data can be found at [https://github.com/kris927b/SkillSpan](https://github.com/kris927b/SkillSpan).

Data AnnotationWe annotate competences related to SKCs in two levels as illustrated in Figure 1. We started the process in March 2021, with initial annotation rounds to construct and refine the annotation guidelines (as outlined further below). The annotation process spanned eight months in total. Our final annotation guidelines can be found in Appendix B. The guidelines were developed by largely following example spans given in the ESCO taxonomy. However, at this stage, we focus on span identification, and we do not take the fine-grained taxonomy codes from ESCO for labeling the spans, leaving the mapping to ESCO and taxonomy enrichment as future work.

Further Details on the Annotation ProcessThe development of the annotation guidelines and our annotation process is depicted as follows: 1 We wrote base guidelines derived from a small number of JPs. 2 We had three pre-rounds consisting of three JPs each. After each round, we modified, improved and finalized the guidelines. 3 Then, we had three longer-lasting annotation rounds consisting of 30 JPs each. We re-annotated the previous 11 JPs in 1 and 2. 4 After these rounds, one of the annotators (the hired linguist) annotated JPs in batches of 50. The data in 1, 2, and 3 was annotated by three annotators (101 JPs).

We used an open source text annotation tool named DoccanoNakayama et al. (2018). There are around 57.5K tokens (approximately 4.6K sentences, in 101 job posts) that we calculated agreement on. The annotations were compared using Cohen's \(\kappa\)Fleiss and Cohen (1973) between pairs of annotators, and Fleiss' \(\kappa\)Fleiss (1971), which generalises Cohen's \(\kappa\) to more than two concurrent annotations. We consider two levels of \(\kappa\) calculations: Token is calculated on the token level, comparing the agreement of annotators on each token (including non-entities) in the annotated dataset. Span refers to the agreement between annotators on the exact span match over the surface string, regardless of the type of SKC, i.e., we only check the position of tag without regarding the type of the entity. The observed agreements scores over the three annotators from step 3 are between 0.70-0.75 Fleiss' \(\kappa\) for both levels of calculation which is considered a _substantial agreement_Landis and Koch (1977) and a \(\kappa\) value greater than 0.81 indicates _almost perfect agreement_. Given the difficulty of this task, we consider the aforementioned \(\kappa\) score to be strong. Particularly, we observed a large improvement in annotation agreement from the earlier rounds (step 1 and 2), where our Fleiss' \(\kappa\) was 0.59 on token-level and 0.62 for the span-level.

Overall, we observe higher annotator agreement for knowledge components (3-5% higher) com

Figure 2: **Violin Plots of Annotated Components.** Indicated are the distributions regarding the length of spans in each type of annotated component (i.e., length of skills and knowledge components). The white dot is the median length, the bars range from the first quartile to the third quartile, and the colored line ranges from the lower adjacent value to the higher adjacent value.

 

pared to skills which tend to be longer. The Tech domain is the most consistent for agreement while Big shows more variation over rounds, likely due to the broader nature of the domains of JPs.

Annotation Span StatisticsA challenge of annotating spans is the length (i.e., boundary), SKCs being in different domains (e.g., business versus technical components), and frequently written differently, e.g., "being able to work together" v.s. "teamwork"). Figure 2 shows the statistics of our annotations in violin plots. For the training set, the median length (white dot) of skills is around 4 for Big and House, for Tech this is a median of 5. In the development set, the median stays at length 4 across all sources. Another notable statistic is the upper and lower percentile of the length of skills and knowledge, indicated with the thick bars. Here, we highlight the fact that skill components could consist of many tokens, for example, up to length 7 in the House source split (see blue-colored violins). For knowledge components, the spans are usually shorter, where it is consistently below 5 tokens (see orange-colored violins). All statistics follow a similar distribution across train, development, and sources in terms of length and distribution. This gives a further strong indication that consistent annotation length has been conducted across splits and sources.

Qualitative Analysis of AnnotationsQualitative differences in SKCs over the three sources are shown (lowercased) in Table 3. With respect to skill components, all sources follow a similar usage of skills. The annotated skills mostly relate to the _attitude_ of a person and hence mostly consist of soft skills. With respect to knowledge components, we observe differences between the three sources. First, on the source-level, the knowledge components vastly differ between Big and Tech. Big postings seem to cover more _business_ related components, whereas Tech has more _engineering_ components. House seems to be a mix of the other two sources. Lastly, note that both the skill and knowledge components between the splits diverge in terms of the type of annotated spans, which indicates a variation in the annotated components. We show the top-10 skills annotated in the train, development, and test splits for SKCs in Appendix C. From a syntactic perspective, skills frequently consist of noun phrases, verb phrases, or adjectives (for soft skills). Knowledge components usually consists of nouns or proper nouns, such as "python", "java", and so forth.

## 5 Experimental Setup

The task of SE is formulated as a sequence labeling problem. Formally, we consider a set of JPs \({\mathcal{D}}\), where \(d\in{\mathcal{D}}\) is a set of sequences (i.e., entire JPs) with the \(i^{\text{th}}\) input sequence \({\mathcal{X}}^{i}_{d}=\{x_{1},x_{2},...,x_{T}\}\) and a target sequence of BIO-labels \({\mathcal{Y}}^{i}_{d}=\{y_{1},y_{2},...,y_{T}\}\) (e.g., "B-SKILL", "I-KNOWLEDGE", "O"). The goal is to use \({\mathcal{D}}\) to train a sequence labeling algorithm \(h:{\mathcal{X}}\mapsto{\mathcal{Y}}\) to accurately predict entity spans by assigning an output label \(y_{t}\) to each token \(x_{t}\).

As baseline we consider BERT and we investigate more recent variants, and we also train models from scratch. Models are chosen due to their state-of-the-art performance, or in particular, for their strong performance on longer spans.

BERTbase[Devlin et al., 2019]An out-of-the-box BERTbase model (bert-base-cased) from the HuggingFace library Wolf et al. (2020) functioning as a baseline.

SpanBERT[Joshi et al., 2020]A BERT-style model that focuses on span representations as opposed to single token representations. SpanBERT is trained by masking contiguous spans of tokens and optimizing two objectives: (1) masked language modeling, which predicts each masked token from its own vector representation. (2) The span boundary objective, which predicts each masked token from the representations of the unmasked tokens at the start and end of the masked span.

We train a SpanBERTbase model from scratch on the BooksCorpus Zhu et al. (2015) and English Wikipedia using cased Wordpiece tokens Wu et al. (2016). We use AdamW Kingma and Ba (2015) for

\begin{table}
\begin{tabular}{l|l l l} \hline  & BIG & HOUSE & TECH \\ \cline{2-4}  & ambitious & structured & hands-on \\  & proactive & tachaging & communication skills \\  & work independently & communication skills & leadership \\  & attention to detail & project management & passionate \\  & motivated & drive & open-minded \\ \hline \multirow{8}{*}{Baselines} & full kit driving licence & english & java \\  & any energy assessments & supply chain & javascript \\ \cline{1-1}  & right to wock in the uk & project management & awes \\ \cline{1-1}  & sen & powders & docker \\ \cline{1-1}  & acc/aca & machine learning & node;js \\ \hline \end{tabular}
\end{table}
Table 3: **Most Frequent Skills in the Development Data. Top-5 skill components in our data in terms of frequency on different sources. A larger example can be found in Table 8 and Table 9 (Appendix C).

 

2.4M training steps with batches of 256 sequences of length 512. The learning rate is warmed up for 10K steps to a maximum value of \(1e\)-4, after which it has a decoupled weight decay [3] of 0.1. We add a dropout rate of 0.1 across all layers. Pretraining was done on a v3-8 TPU on the GCP and took 14 days to complete. We take the official TensorFlow implementation of SpanBERT by ram2021bert.

**JobBERT7** We apply domain-adaptive pre-training [9] to a BERTbase model using the 3.2M unlabeled JP sentences (Table 2). Domain-adaptive pre-training relates to the continued self-supervised pre-training of a large language model on domain-specific text. This approach improves the modeling of text for downstream tasks within the domain. We continue training the BERT model for three epochs (default in HuggingFace) with a batch size of 16.

Footnote 7: [https://huggingface.co/jjzha/jobbert-base-cased](https://huggingface.co/jjzha/jobbert-base-cased)

**JobSpanBERT8** We apply domain-adaptive pre-training to our SpanBERT on 3.2M unlabeled JP sentences. We keep parameters identical to the vanilla SpanBERT, but change the number of steps to 40K to have three passes over the unlabeled data.

Footnote 8: [https://huggingface.co/jjzha/jobspanbert-base-cased](https://huggingface.co/jjzha/jobspanbert-base-cased)

ExperimentsWe have 391 annotated JPs (Table 2) that we divide across three splits: Train, dev. and test set. We use 101 JPs that all three annotators annotated as the gold standard test set with aggregated annotations via majority voting. The 101 postings are divided between the sources as: 36 Big, 33 House, and 32 Tech. The remaining 290 JPs were annotated by one annotator. We use 90 JPs (30 from each source, namely Big, House, and Tech) as the dev. set. The remaining 200 JPs are used as the train set. The sources in the train set are divided into 60 Big, 60 House, and 80 Tech.

**Setup** The data is structured as CoNLL format [10]. For the nested annotations, the skill tags are appearing only in the first column and the knowledge tags are only appearing in the second column of the file and they are allowed to overlap with each other. We perform experiments with single-task learning (STL) on either the skill or knowledge components, MTL for predicting both skill and knowledge tags at the same time, while evaluating the MTL models also on either skills or knowledge components. We used a single joint MTL model with hard-parameter sharing [9]. All models are with a final Conditional Random Field (CRF; Lafferty et al.,

Figure 3: **Performance of Models. We test the models on Skills, Knowledge, and Combined. We report the span-F1 and standard deviation (error bars) of runs on five random seeds. Note that the y-axis starts from 50 span-F1. STL indicates single-task learning and MTL indicates the multi-task model. Differences can be seen on the test set: JobSpanBERT performs best on Skills, JobBERT is best on Knowledge, and JobBERT achieves best in Combined. Exact numbers of the plots are in Table 5 (Appendix E).

 

2001) layer. Earlier research, such as Souza et al. (2019); Jensen et al. (2021) show that BERT models with a CRF-layer improve or perform similarly to its simpler variants when comparing the overall F1 and make no tagging errors (e.g., B-tag follows I-tag). In the case of MTL we use one for each tag type (skill and knowledge). In the STL experiments we use one CRF for the given tag type.

We use the MaChAmp toolkit van der Goot et al. (2021) for our experiments. For each setup we do five runs (i.e., five random seeds).9 For evaluation we use span-level precision, recall, and F1, where the F1 for the MTL setting is calculated as described in Benikova et al. (2014).

Footnote 9: For reproducibility, we refer to Appendix D.

## 6 Results

The results of the experiments are given in Figure 3. We show the average performance of each model in F1 and respective standard deviation over the development and test split. Exact scores on each source split and other metric details are provided in Appendix E. As mentioned before, we experiment with the following settings: Skill, we train and predict only on skills. Knowledge, train and only predict for knowledge. Combined, we merge the STL predictions of both skills and knowledge. We also train the models in an MTL setting, predicting both skills and knowledge simultaneously. We evaluate the MTL model on both Skill and Knowledge separately, and also compare it against the aggregated STL predictions.

Performance on Development SetIn Figure 3, we show the results on the development set in the upper plot. We observe similar performance between the domain-adapted STL models--JobBERT and JobSpanBERT--have similar span-F1 for Skill: 60.05\(\pm\)0.70 vs. 60.07\(\pm\)0.70. In contrast, for Knowledge, BERTbase and JobBERT are closest in predictive performance: 60.44\(\pm\)0.58 vs. 60.66\(\pm\)0.43. In the Combined setting, JobBERT performs highest with a span-F1 of 60.32\(\pm\)0.39. On average, JobBERT performs best over all three settings. Surprisingly, the models for both Skill and Knowledge perform similarly (around 60 span-F1), despite the sources' differences in properties and length Figure 2. In addition, we find that MTL is not performing better than STL across sources. For exact numbers and source-level (i.e., Big, House, Tech), we refer to Appendix E.

Performance on Test SetWe select the best performing models in the development set evaluation and apply it to the test set. Results are in Figure 3 in the bottom plot. Since JobBERT and JobSpanBERT are performing similarly, we apply both to the test set and BERTbase. We observe a deviation from the development set to the test set: JobSpanBERT 60.07\(\pm\)0.30\(\rightarrow\)56.64\(\pm\)0.83 on Skill, JobBERT 60.66\(\pm\)0.43\(\rightarrow\)63.88\(\pm\)0.28 on Knowledge. For Combined, JobBERT performs slightly worse: 60.32\(\pm\)0.39\(\rightarrow\)59.73\(\pm\)0.38. Similar to the development set, we find that on all three methods of evaluation (i.e., Skill, Knowledge, and Combined), STL still outperforms MTL. For Skill and Knowledge, STL is almost stochastically dominant over MTL (i.e., significant), and for Combined there is stochastic dominance of STL over MTL, indicated in the next paragraph.

SignificanceWe compare all pairs of models based on five random seeds each using Almost Stochastic Order (ASO; Dror et al., 2019) tests with a confidence level of \(\alpha=0.05\). The ASO scores of the test set are indicated in Figure 4. We show that MTL-JobSpanBERT for Skill shows

Figure 4: **Almost Stochastic Order Scores of the Test Set. ASO scores expressed in \(\epsilon_{\min}\). The significance level \(\alpha=0.05\) is adjusted accordingly by using the Bonferroni correction Bonferroni (1936). Read from row to column: E.g., in Combined STL-JobBERT (row) is stochastically dominant over STL-BERTbase (column) with \(\epsilon_{\min}\) of 0.00.**



almost stochastic dominance (\(\epsilon_{\text{min}}<0.5\)) over all other models. For Knowledge and Combined, We show that STL-JobBERT is stochastically dominant (\(\epsilon_{\text{min}}=0.0\)) over _all_ the other models. For more details, we refer to Appendix F for ASO scores on the development set.

## 7 Discussion

What Did Not WorkAdditionally, we experiment whether representing the entire JP for extracting tokens yields better results than the experiments so far, which were sentence-by-sentence processing setups. To handle entire JPs and hence much longer sequences we use a pre-trained LongformerbaseBeltagy et al. (2020) model. The document length we use in the experiments is 4096 tokens. Results of the Longformer on the test set are lower: For skills, JobSpanBERT against Longformer results in 56.64\(\pm\)0.83 vs. 52.55\(\pm\)2.39. For Knowledge, JobBERT against Longformer shows 63.88\(\pm\)0.28 vs. 57.26\(\pm\)1.05. Last, for Combined, JobBERT against Longformer results in 59.73\(\pm\)0.38 vs. 55.05\(\pm\)0.71. This drop in performance is difficult to attribute to a concrete reason: e.g., the Longformer is trained on more varied sources than BERT, but not specifically for JPs, which may have contributed to this gap. Since the vanilla Longformer already performs worse than BERTbase overall, we did not opt to apply domain-adaptive pre-training. Overall, we show that representing the full JP is not beneficial for SE, at least not in the Longformer setup tested here.

Continuous Pretraining helps SEAs previously mentioned, due to the domain specialization of the domain-adapted pre-trained BERT models, they predict more skills and frequently perform better in terms of precision, recall, and F1 as compared to their non-adaptive counterparts. This is especially encouraging as we confirm findings that continuous pre-training helps to adapt models to a specific domain Alsentzer et al. (2019); Lee et al. (2020); Gururangan et al. (2020); Nguyen et al. (2020). However, there are exceptions. Particularly in Table 5 on Test for Knowledge, BERTbase comes closer in predictive performance to JobBERT (difference of 1.5 F1) than on Skills. Our intuition is that knowledge components are often already in the pre-training data (e.g., Wikipedia pages of certain competences like Python, Java etc.) and therefore adaptive pre-training does not substantially boost performance.

Difference in Length of PredictionsThe main motivation of selecting models optimized for long spans was the length of the annotations (Figure 2). We investigate the average length of predictions of each model (Figure 5) to find out whether the models that are adapted to handle longer sequences truly predict longer spans. Interestingly, the average length of predicted skills are longer than the annotations over all three sources. There is a consistent trend among Skill: Big and Tech have similar length over predictions (>4), while House is usually lower than length 3. For both Big and Tech, JobSpanBERT predicts the longest skill spans (4.51 and 4.48 respectively). We suspect due to the domain-adaptive pre-training on JPs, it improved the span prediction performance. In contrast, the Longformer predicts shorter spans. Note that the Longformer is not domain-adapted to JPs.

Regarding Knowledge, there is also a consistent trend: Big has the overall longest prediction length while Tech has the lowest. The Longformer predicts the longest spans on average for Big and Tech. Knowledge components are representative of a normal-length NER task and might not need a specialized model for long sequences. We show the exact numbers in Table 7 (Appendix E) and the number of predicted Skill and Knowledge: JobBERT and JobSpanBERT have higher recall than the other models.

Performance per Span LengthSkills are generally longer than Knowledge components in our dataset (Figure 2). The previous overall results on the test set (Figure 3) show that performance on

Figure 5: **Average Length of Predictions of Single Models.** We show the average length of the predictions versus the length of our annotated skills and knowledge components on the _test set_ and the total number of predicted skills and knowledge tags in each respective split (#). There is a consistent trend over the three sources.

 

Skill is substantially lower than Knowledge. We therefore investigate whether this performance difference is attributed to the longer spans in Skill. In Figure 6, we show the average performance of the best performing model (JobBERT) on the three sources (test set) based on the gold span length, until a length of 10.

In Skill components (upper plot), we see much support for spans with length 1 and 2, which then lowers once the spans become longer. Spans with length of 1 shows low performance on Big and Tech (around 40 span-F1), which influences the total span-F1. Short skills are usually soft skills, such as "passionate", which can be used as a skill or not. This might confuse the model. In contrast, performance effectively stays similar (around 60 span-F1) for span length of 2 till 7 for all sources. Afterwards, it drops in performance. Thus, the weak performance on Skill seem to be due to lower performance on the short spans.

For the Knowledge components (lower plot), they are generally shorter. We see that there is a gap in support between the sources, Tech has a larger number of gold labels compared to Big and House. Unlike soft skills, KCs usually consist of proper nouns such as "Python", "Java", and so forth, which connects to the high performance on Tech (around 76 span-F1). Furthermore, support for spans longer than 2 drops considerably. In this case, if the model predicts a couple of instances correctly, it would substantially increase span-F1. Contrary to Skill, high performance of Knowledge can be attributed to its strong performance on short spans.

## 8 Conclusion

We present a novel dataset for skill extraction on English job postings-- SkillSpan--and domain-adapted BERT models--JobBERT and JobSpanBERT. We outline the dataset and annotation guidelines, created for hard _and_ soft skills annotation on the _span-level_. Our analysis shows that domain-adaptive pre-training helps to improve performance on the task for both skills and knowledge components. Our domain-adapted JobSpanBERT performs best on skills and JobBERT on knowledge. Both models achieve almost stochastic dominance over all other models for skills and knowledge extraction, whereas JobBERT in the STL setting achieves stochastic dominance over other models.

With the rapid emergence of new competences, our new approach of skill extraction has future potential, e.g., to enrich knowledge bases such as ESCO with unseen skills or knowledge components, and in general, contribute to providing insights into labor market dynamics. We hope our dataset encourages research into this emerging area of computational job market analysis.

Figure 6: **Average Span-F1 per Span Length. We bucket the performance of JobBERT according to the length of the spans until 10 tokens and show the performance on each length, averaged over five random seeds. Indicated per bar is the support. The model performs best on medium-length skill spans (i.e., spans with token length of 4-5). For knowledge spans, on average, it performs best on short-length spans (i.e., spans with token length of 1-2).**





## Acknowledgements

We thank Google's TFRC for their support in providing TPUs for this research. Furthermore, we thank the NLPnorth group for feedback on an earlier version of this paper--in particular, Elisa Bassignana and Max Muller-Eberstein for insightful discussions. We would also like to thank the anonymous reviewers for their comments to improve this paper. Last, we also thank NVIDIA and the ITU High-performance Computing cluster for computing resources. This research is supported by the Independent Research Fund Denmark (DFF) grant 9131-00019B.

## References

* A. Alsentzer, J. Murphy, W. Boag, W. Weng, D. Jindi, T. Naumann, and M. McDermott (2019)Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, Minnesota, USA, pp. 72-78. External Links: Link Cited by: SS4.1.
* K. Balog, Y. Fang, M. De Rijke, P. Serdyukov, and L. Si (2012)Expertise retrieval. Foundations and Trends in Information Retrieval6 (2-3), pp. 127-256. External Links: Document, ISSN 1573-030X Cited by: SS1.
* I. Beltagy, M. E Peters, and A. Cohan (2020)Longformer: The long-document transformer. arXiv preprintabs/2004.05150. Cited by: SS1.
* E. M. Bender and B. Friedman (2018)Data statements for natural language processing: toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics6, pp. 587-604. External Links: Link, Document Cited by: SS1.
* D. Benikova, C. Biemann, and M. Reznicek (2014)NoSta-d named entity annotation for German: guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland, pp. 2524-2531. External Links: Link, ISBN 978-3-319

* Hochreiter et al. (1997) Sepp Hochreiter, Jurgen Schmidhuber, and Corso Elvezia. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Javed et al. (2017) Faizan Javed, Phuong Hoang, Thomas Mahoney, and Matt McNair. 2017. Large-scale occupational skills normalization for online recruitment. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_, pages 4627-4634. AAAI Press.
* Jensen et al. (2021) Kristian Norgaard Jensen, Mike Zhang, and Barbara Plank. 2021. De-identification of privacy-related entities in job postings. In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_, pages 210-221, Reykjavik, Iceland (Online). Linkoping University Electronic Press, Sweden.
* Jia et al. (2018) Shanshan Jia, Xiaoan Liu, Ping Zhao, Chang Liu, Lianging Sun, and Tao Peng. 2018. Representation of jobskill in artificial intelligence with knowledge graph analysis. In _2018 IEEE Symposium on Product Compliance Engineering-Asia (ISPCE-CN)_, pages 1-6. IEEE.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77.
* Khaouja et al. (2021) Imane Khaouja, Ismail Kassou, and Mounir Ghogho. 2021. A survey on skill identification from online job ads. _IEEE Access_, 9:118134-118153.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kivimaki et al. (2013) Ilkka Kivimaki, Alexander Panchenko, Adrien Dessy, Dries Verdegem, Pascal Francq, Hugues Bersini, and Marco Saerens. 2013. A graph-based approach to skill extraction from text. In _Proceedings of TextGraphs-8 Graph-based Methods for Natural Language Processing_, pages 79-87, Seattle, Washington, USA. Association for Computational Linguistics.
* July 1, 2001_, pages 282-289. Morgan Kaufmann.
* Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. _biometrics_, pages 159-174.
* Velrang et al. (2014) Martin le Vrang, Agis Papantoniou, Erika Pauwels, Pieter Fannes, Dominique Vandenstein, and Johan De Smedt. 2014. Esco: Boosting job matching in europe with semantic interoperability. _Computer_, 47(10):57-64.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Li et al. (2020) Shan Li, Baoxu Shi, Jaewon Yang, Ji Yan, Shuai Wang, Fei Chen, and Qi He. 2020. Deep job understanding at linkedin. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_, pages 2145-2148. ACM.
* Liu et al. (2021)Liting Liu, Wenzheng Zhang, Jie Liu, Wenxuan Shi, and Yalou Huang. 2021. Learning multi-graph neural network for data-driven job skill prediction. In _2021 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from [https://github.com/doccano/doccano](https://github.com/doccano/doccano).
* Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 9-14, Online. Association for Computational Linguistics.
* Ram et al. (2021) Ori Ram, Yuval Kirstan, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3066-3079, Online. Association for Computational Linguistics.
* Reimers and Gurevych (2018) Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. _ArXiv preprint_, abs/1803.09578.
* Sayfullina et al. (2018) Luiza Sayfullina, Eric Malmi, and Juho Kannala. 2018. Learning representations for soft skill matching. In _International Conference on Analysis of Images, Social Networks and Texts_, pages 141-152.
* Shi et al. (2020) Baoxu Shi, Jaewon Yang, Feng Guo, and Qi He. 2020. Salience and market-aware skill extraction for job targeting. In _KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_, pages 2871-2879. ACM.

 

Ellery Smith, Martin Braschler, Andreas Weiler, and Thomas Haberthuer. 2019. Syntax-based skill extractor for job advertisements. In _2019 6th Swiss Conference on Data Science (SDS)_, pages 80-81. IEEE.
* Smith et al. [2021] Ellery Smith, Andreas Weiler, and Martin Braschler. 2021. Skill extraction for domain-specific text retrieval in a job-matching platform. In _International Conference of the Cross-Language Evaluation Forum for European Languages_, pages 116-128. Springer.
* Souza et al. [2019] Fabio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2019. Portuguese named entity recognition using bert-crf. _ArXiv preprint_, abs/1909.10649.
* Tamburri et al. [2020] Damian A Tamburri, Willem-Jan Van Den Heuvel, and Martin Garriga. 2020. Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In _2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI)_, pages 391-394. IEEE.
* Tjong Kim Sang [2002] Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Ulmer [2021] Dennis Ulmer. 2021. deep-significance: Easy and Better Significance Testing for Deep Neural Networks. [https://github.com/Kaleidophon/deep-significance](https://github.com/Kaleidophon/deep-significance).
* van der Goot et al. [2021] Rob van der Goot, Ahmet Ustun, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_, pages 176-197, Online. Association for Computational Linguistics.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _ArXiv preprint_, abs/1609.08144.
* Zhao et al. [2015] Meng Zhao, Faizan Javed, Ferosh Jacob, and Matt McNair. 2015. SKILL: A system for skill identification and normalization. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_, pages 4012-4018. AAAI Press.
* Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 19-27. IEEE Computer Society.

 

Data Statement SkillsSpan

Following Bender and Friedman (2018), the following outlines the data statement for SkillSpan:

* Curation RationaleCollection of job postings in the English language for span-level sequence labeling, to study the impact of sequence labeling on the extraction of skill and knowledge components from job postings.
* Language VarietyThe non-canonical data was collected from the StackOverflow job posting platform, an in-house job posting collection from our national labor agency collaboration partner (_which will be elaborated upon acceptance_), and web extracted job postings from a large job posting platform. US (en-US) and British (en-GB) English are involved.
* Speaker DemographicGender, age, raceethnicity, socioeconomic status are unknown.
* Annotator DemographicThree hired project participants (age range: 25-30), gender: one female and two males, white European and Asian (non-Hispanic). Native language: Danish, Dutch. Socioeconomic status: higher-education students. Female annotator is a professional annotator with a background in Linguistics and the two males with a background in Computer Science.
* Speech SituationStandard American or British English used in job postings. Time frame of the data is between 2012-2021.
* Text CharacteristicsSentences are from job postings posted on official job vacancy platforms.
* Recording QualityN/A.
* OtherN/A.

I. Provenance AppendixThe job posting data in Techis from Stackoverflow jobs, and is licensed under the CC BY-SA license. The job posting data from House is from our collaborators: The Danish Agency for Labour Market and Recruitment (STAR).

 



## Appendix B Annotation Guidelines

### Span Specifications

**Legend:** Skill, Knowledge, "*" indicates an example sentence.

1. A skill starts with a **VERB**, otherwise (**ADJECTIVE**) + **NOUN**

1.1 Modal verbs are not tagged:

* Can [put personal touch on the menu]\({}_{\text{skill}}\).
* Will [train new staff]\({}_{\text{skill}}\).

2. Split up phrases with prepositions and/or conjunctions

2.1 **Unless** the conjunction coordinates two nouns functioning as one argument:

* Coordinate parties and conferences]\({}_{\text{skill}}\).

2.2 Do not tag skills with anaphoric pronouns, only tag preceding skill:

* Prioritizing tasks]\({}_{\text{skill}}\) and identifying those that are most important.

2.3 Split nouns and adjectives that are coordinated if they do not have a verb attached:

* Be [inquisitive]\({}_{\text{skill}}\) and [proactive]\({}_{\text{skill}}\).
* Prior in-house experience with [media]\({}_{\text{skowlledge}}\), [publishing]\({}_{\text{skowlledge}}\) or [internet companies]\({}_{\text{skowlledge}}\).

2.4 If there is a listing of skill tags and they lead up to different subtasks, we split them:

* [keep up the high level of quality in our team]\({}_{\text{skill}}\) through [reviews]\({}_{\text{skill}}\), [pairing]\({}_{\text{skill}}\) and [mentoring]\({}_{\text{skill}}\).

3. If there is relevant information appended after irrelevant information (e.g., info specific to a company) we try to make the skill as short as possible:

* [providing the best solution]\({}_{\text{skill}}\) for--Siemens--Gamesa--in--every [structured]\({}_{\text{skilled}}\) and [analytic]\({}_{\text{skill}}\) manner.

4. Note also the words skills and knowledge can be included in the span of the component if leaving it out makes it nonsensical:

* [personal skills]\({}_{\text{skill}}\)\(\rightarrow\) just [personal] would make it nonsensical.

5. Parentheses after a skill tag are included if they elaborate the component before them or if they are an abbreviation of the component.



6. **Inclusion of adverbials in components**. Adverbials are included if it concerns the manner of doing something. All others are excluded:

* like-to solve technical challenges independently\({}_{\text{skill}}\).
* communicates openly\({}_{\text{skill}}\).
* [striving for the best]\({}_{\text{skill}}\) in all that they do.
* Deliver first class customer service]\({}_{\text{skill}}\) to our guests.
* Making the right decisions]\({}_{\text{skill}}\) early in the process.

7. **Attitudes as skills.** We annotate attitudes as a skill:

* a [can-do-approach]\({}_{\text{skill},\bot}\)\(\rightarrow\) we leave out articles from the attitude.

8. Attitudes are not tagged if they contain skill/knowledge components--then only the span of the skill is tagged.

* like-to solve technical challenges independently\({}_{\text{skill}}\).
* Passion-for [automation]\({}_{\text{kNOWLEDG}}\).
* enjoy Working in a team]\({}_{\text{skill}}\).

9. **Miscellaneous:**

9.1 Do not tag ironic skills (e.g., lazy).

9.2 Avoid nesting of skills, annotate it as one span.

9.3 We annotate all skills that are part of sections such as "requirements", "good-to-haves", "great-to-knows", "optionals", "after this \(x\) months of training you'll be able to...", "At the job you're going to...".

9.4 When there is a general standard that can be added to the skill, we add these:

* [Process payments according to the [...] standards]\({}_{\text{skill}}\).

 



### Knowledge Specifications

1. **Rule-of-thumb**: knowledge is something that one possesses, and cannot (usually) physically execute:
2. If there is a component between parentheses that belongs to the knowledge component, we add it:
3. [non-) relational databases]\({}_{\text{knownLEDGE}}\).
4. [Driver License (UK/EU)]\({}_{\text{knownLEDGE}}\).

3. **Licensees and certifications**: We add the additional words "certificate", "card", "license", et cetera. to the knowledge component.

4. If the knowledge component looks like a skill, but the preceding verb is vague and empty (e.g., _follow_, _use_, _comply with_, _work with_) only tag the knowledge component:

* Comply-with [Food Code of Practice]\({}_{\text{knownLEDGE}}\).
* Work-with [AWS infrastructure]\({}_{\text{knownLEDGE}}\).

5. We annotate only specified knowledge components:
6. [MongoDB]\({}_{\text{knownLEDGE}}\) or other [NoSQL database]\({}_{\text{knownLEDGE}}\).
7. If all components coordinate/share one knowledge tag, we annotate it as one:
8. [application, data and infrastructure architecture]\({}_{\text{knownLEDGE}}\). \(\rightarrow\) The knowledge tags coordinate to "architecture".
9. [chemical/biochemical engineering]\({}_{\text{knowLEDGE}}\).

8. If there is a listing of knowledge tags, we annotate all knowledge tags separately:

* [Bachelor Degree]\({}_{\text{knownLEDGE}}\) in [Mathematics]\({}_{\text{knownLEDGE}}\), [Computer Science]\({}_{\text{knownLEDGE}}\), or [Engineering]\({}_{\text{knownLEDGE}}\).





### Other Specifications

1. **Rule-of-thumb**: If in doubt, annotate it as a skill.
2. We are preferring skills over knowledge components.
3. We prioritize skills over attitudes; if there is a skill within the attitude, only tag the skill:
4. Skills or knowledge components in the top headlines of the JP are not tagged (e.g., title of a JP). If it is a sub-headline or in the rest of the posting, tag it.
5. We try to keep the skill/knowledge components as short as possible (i.e., exclude information at the end if it makes it too specific for the job).
6. We do not include "fluff" and "triggers" (i.e., words that indicate a skill or knowledge component will follow: "advanced-knowledge-of [...]\({}_{\text{knownLEDGE}}\)") around the components, including degree. This goes for both before and after:

* Working proficiency in developmental toolsets\({}_{\text{knowLEDGE}}\).
* Advanced-knowledge-of application data and architecture infrastructure\({}_{\text{knowLEDGE}}\) disciplines.
* Manual handling\({}_{\text{skille}}\) tasks.
* CI/CD\({}_{\text{KNOWLEDGE}}\) experience.
* You-master English\({}_{\text{knowLEDGE}}\) on-level Ct.
* Proficient-in [Python]\({}_{\text{KnowLEDGE}}\) and [English]\({}_{\text{knowLEDGE}}\).
* Fluent in spoken and written [English]\({}_{\text{knowLEDGE}}\).
7. Pay attention to expressions such as "participation in...", "contributing", and "transfer (knowledge)". These are usually not considered skills.

* Contribute to the enjoyable and collaborative work environment.
* Participation in the Department's regular research activities.
* Desire to be part of something meaningful and innovative.
8. Skills and Knowledge components that are found in not-so-straightforward places (e.g., project descriptions) are annotated as well, if they relate to the position.
9. In the pattern of "skill" followed by some elaboration, see if it can be annotated with a skill and a knowledge tag:

* [Ensure food storage and preparation areas are maintained]\({}_{\text{skille}}\) according to Health & Safety and Audit standards\({}_{\text{KNOWLEDGE}}\).



10. Occupations and positions in companies/academia should be excluded.

11. If there's a knowledge/skill component in the position, we exclude it as well.

* Experienced Java Engineer. \(\rightarrow\) completely untagged.

12. Only annotate the skills that are related to the position.

12.1. This includes skills that are specific for the position as well (e.g., skills of a ruminants professor versus math professor).

12.2 Also skills that the person for the position is expected to do in the future.

12.3 This does not include skills, knowledge or attitudes describing only the company, the group you will join in the department, and so on. Only annotate if it is specified or implied that the employee should possess the skill as well.

13. We annotate industries and fields (that the employee will be working in) as knowledge components.

 

Type of Skills Annotated

In both Table 8 and Table 9, we show the top-10 skill and knowledge components that have been annotated. We split the top-10 among the data splits (i.e., train, development, and test set), and also between source splits (i.e., Big, House, Tech).

## Appendix D Reproducibility

We use the default hyperparameters in MaChaAmp van der Goot et al. (2021) as shown in Table 4. For more details we refer to their paper. For the five random seeds we use 3477689, 4213916, 6828303, 8749520, and 9364029. All experiments with MaChAmp were ran on an NVIDIA(r) TITAN X (Pascal) 12 GB GPU and an Intel(r)Xeon(r) Silver 4214 CPU.

## Appendix E Exact Number of Performance

In Table 5, we show the exact numbers of the plot indicated in Figure 3. In addition, we also show the results of each respective split.

For the STL models, we observe differences in performances over the sources which is particularly pronounced for knowledge components: The Tech source is the easiest to process (and has most SKCs), while SKCs identification performance is the lowest for Big. This might be due to the broad nature of this source.

In the exact results table (Table 5) we add a (\({\dagger}\)) next to the highest span-F1 if the model is truly stochastically dominant (\(\epsilon_{\text{min}}=0.0\)) over _all_ the other models. (*) denotes that the best model achieved _almost stochastic dominance_ (\(\epsilon_{\text{min}}<0.5\)) over--at minimum--one other model (e.g., in Test rows w.r.t Combined: MTL-JobBERT \(\succcurlyeq\) MTL-JobSpanBERT with \(\epsilon_{\text{min}}=0.06\)) and stochastically dominant over the rest.

In Table 6, we report the precision and recall of the models, Skill and Knowledge show the precision and recall of the STL models. Multi shows the precision and recall of the MTL models.

Last, in Table 7, we show the exact numbers of the length of predictions Figure 5. We also add the number of predicted Skill and Knowledge Overall, JobBERT and JobSpanBERT predict more skills in general than the other models. This is also the case for knowledge components. We hypothesize that this might be due to the BERT models now being more specialized towards the JP domain and recognizing more SKCs.

## Appendix F Significance Testing

Recently, the ASO test Dror et al. (2019)10 has been proposed to test statistical significance for deep neural networks over multiple runs. Generally, the ASO test determines whether a stochastic order Reimers and Gurevych (2018) exists between two models or algorithms based on their respective sets of evaluation scores. Given the single model scores over multiple random seeds of two algorithms \(\mathcal{A}\) and \(\mathcal{B}\), the method computes a test-specific value (\(\epsilon_{\text{min}}\)) that indicates how far algorithm \(\mathcal{A}\) is from being significantly better than algorithm \(\mathcal{B}\). When distance \(\epsilon_{\text{min}}=0.0\), one can claim that \(\mathcal{A}\) stochastically dominant over \(\mathcal{B}\) with a predefined significance level. When \(\epsilon_{\text{min}}<0.5\) one can say \(\mathcal{A}\succeq\mathcal{B}\). On the contrary, when we have \(\epsilon_{\text{min}}=1.0\), this means \(\mathcal{B}\succeq\mathcal{A}\). For \(\epsilon_{\text{min}}=0.5\), no order can be determined. We took 0.05 for the predefined significance level. In Figure 7, we show the ASO scores on the development set.

Footnote 10: Implementation of Dror et al. (2019) can be found at [https://github.com/Kaleidophon/deep-significanceUlmer](https://github.com/Kaleidophon/deep-significanceUlmer) (2021) 



\begin{table}
\begin{tabular}{l|c|c c|c c|c c} \hline  & **Evaluation**\(\rightarrow\) & \multicolumn{2}{c|}{**Skill**} & \multicolumn{2}{c|}{**Knowledge**} & \multicolumn{2}{c}{**Multi**} \\ \cline{3-8} \multicolumn{1}{c|}{**Src.**} & \(\downarrow\)**Model** & **Precision** & **Recall** & **Precision** & **Recall** & **Precision** & **Recall** \\ \hline \multirow{6}{*}{**ResNet**} & BERTbase & 57.09\(\pm\)1.70 & 62.27\(\pm\)1.28 & 43.95\(\pm\)4.17 & 60.00\(\pm\)1.65 & 52.63\(\pm\)1.32 & 62.19\(\pm\)0.87 \\  & SpanBERT & 58.28\(\pm\)0.59 & 61.36\(\pm\)0.68 & 45.80\(\pm\)2.89 & 56.82\(\pm\)3.39 & 54.02\(\pm\)1.81 & 62.63\(\pm\)2.60 \\  & JobBERT & 57.90\(\pm\)1.25 & 63.59\(\pm\)0.99 & 43.45\(\pm\)1.98 & 59.84\(\pm\)3.44 & 51.13\(\pm\)0.48 & 63.74\(\pm\)0.79 \\  & JobSpanBERT & 58.39\(\pm\)1.03 & 62.09\(\pm\)1.85 & 38.55\(\pm\)3.12 & 54.76\(\pm\)3.18 & 52.22\(\pm\)0.35 & 61.75\(\pm\)1.22 \\ \cline{2-8}  & \multirow{3}{*}{**ResNet**} & BERTbase & 55.95\(\pm\)2.46 & 57.79\(\pm\)0.67 & 52.84\(\pm\)0.65 & 57.42\(\pm\)2.76 & 51.65\(\pm\)1.11 & 59.28\(\pm\)2.07 \\  & SpanBERT & 56.70\(\pm\)1.59 & 58.44\(\pm\)1.16 & 49.87\(\pm\)2.57 & 54.49\(\pm\)3.09 & 52.27\(\pm\)0.64 & 58.25\(\pm\)1.50 \\  & JobBERT & 58.16\(\pm\)1.30 & 61.56\(\pm\)1.53 & 51.18\(\pm\)2.18 & 59.37\(\pm\)1.34 & 53.72\(\pm\)1.57 & 62.56\(\pm\)1.47 \\  & JobSpanBERT & 59.04\(\pm\)0.85 & 60.99\(\pm\)2.58 & 51.36\(\pm\)2.70 & 60.84\(\pm\)1.19 & 53.91\(\pm\)0.77 & 60.79\(\pm\)0.54 \\ \cline{2-8}  & \multirow{3}{*}{**ResNet**} & BERTbase & 58.28\(\pm\)1.30 & 59.89\(\pm\)1.39 & 60.79\(\pm\)1.89 & 67.79\(\pm\)1.20 & 58.19\(\pm\)1.12 & 65.55\(\pm\)0.75 \\  & SpanBERT & 58.62\(\pm\)0.32 & 58.16\(\pm\)0.76 & 59.43\(\pm\)1.21 & 66.35\(\pm\)1.18 & 58.34\(\pm\)0.97 & 65.17\(\pm\)1.41 \\  & JobBERT & 58.81\(\pm\)1.38 & 60.88\(\pm\)1.51 & 61.38\(\pm\)1.11 & 68.14\(\pm\)1.36 & 57.69\(\pm\)0.93 & 66.25\(\pm\)0.90 \\  & JobSpanBERT & 59.86\(\pm\)3.07 & 60.40\(\pm\)0.68 & 59.78\(\pm\)2.43 & 67.57\(\pm\)1.97 & 58.26\(\pm\)0.82 & 65.82\(\pm\)0.91 \\ \hline \multirow{6}{*}{**ResNet**} & BERTbase & 57.11\(\pm\)1.65 & 59.90\(\pm\)0.95 & 56.86\(\pm\)1.33 & 64.54\(\pm\)1.31 & 55.02\(\pm\)0.85 & 62.98\(\pm\)0.93 \\  & SpanBERT & 57.85\(\pm\)0.65 & 59.23\(\pm\)0.52 & 55.65\(\pm\)1.09 & 62.58\(\pm\)1.56 & 55.61\(\pm\)0.61 & 62.58\(\pm\)1.25 \\  & JobBERT & 58.29\(\pm\)1.08 & 61.94\(\pm\)1.16 & 56.73\(\pm\)1.41 & 65.22\(\pm\)1.03 & 55.03\(\pm\)0.84 & 64.62\(\pm\)0.77 \\  & JobSpanBERT & 59.11\(\pm\)1.59 & 61.12\(\pm\)1.49 & 55.11\(\pm\)2.41 & 64.66\(\pm\)1.38 & 55.64\(\pm\)0.56 & 63.46\(\pm\)0.69 \\ \hline \hline \multirow{6}{*}{**ResNet**} & BERTbase & 56.02\(\pm\)1.50 & 52.79\(\pm\)1.18 & 59.09\(\pm\)0.85 & 66.20\(\pm\)1.69 & 55.82\(\pm\)1.03 & 59.79\(\pm\)0.87 \\  & JobBERT & 55.94\(\pm\)1.19 & 56.29\(\pm\)0.49 & 60.03\(\pm\)1.13 & 68.30\(\pm\)1.46 & 55.87\(\pm\)0.29 & 62.89\(\pm\)0.56 \\ \cline{1-1}  & JobSpanBERT & 57.57\(\pm\)1.24 & 55.77\(\pm\)1.65 & 57.83\(\pm\)1.03 & 64.71\(\pm\)2.10 & 57.06\(\pm\)0.74 & 60.89\(\pm\)0.42 \\ \hline \end{tabular}
\end{table}
Table 6: **Precision and Recall of Models**. We test the models on _skills_, _knowledge_, and _multi-task_ setting. We report the average precision, recall and standard deviation of runs on five random seeds on the _development set_ (**Average**). Results on the _test set_ are below in the **Test** rows.

\begin{table}
\begin{tabular}{l|l|l l|l l|l l} \hline  & **Evaluation**\(\rightarrow\) & \multicolumn{2}{c|}{**Skill**} & \multicolumn{2}{c|}{**Knowledge**} & \multicolumn{2}{c}{**Combined**} \\ \hline
**Src.** & \(\downarrow\)**Model**, **Task**\(\rightarrow\) & **STL** & **MTL** & **STL** & **MTL** & **STL (***2)** & **MTL** \\ \hline \multirow{6}{*}{**ResNet**} & BERTbase & 59.55\(\pm\)0.97 & 58.88\(\pm\)1.14 & 50.68\(\pm\)3.25 & 51.10\(\pm\)1.67 & 57.46\(\pm\)1.19 & 57.00\(\pm\)0.91 \\  & SpanBERT & 59.78\(\pm\)0.44 & 60.02\(\pm\)2.15 & 50.65\(\pm\)2.32 & 51.79\(\pm\)1.22 & 57.71\(\pm\)0.53 & 58.00\(\pm\)2.07 \\  & JobBERT & 60.60\(\pm\)0.81 & 59.76\(\pm\)0.60 & 50.29\(\pm\)1.86 & 47.59\(\pm\)1.11 & 58.19\(\pm\)0.49 & 56.75\(\pm\)0.50 \\  & JobSpanBERT & 60.16\(\pm\)0.61 & 59.44\(\pm\)1.11 & 45.20\(\pm\)2.76 & 47.69\(\pm\)3.38 & 56.56\(\pm\)0.49 & 56.58\(\pm\)0.63 \\ \hline \multirow{6}{*}{**ResNet**} & BERTbase & 56.83\(\pm\)1.29 & 55.89\(\pm\)1.90 & 55.00\(\pm\)1.11 & 54.05\(\pm\)1.00 & 56.17\(\pm\)0.92 & 55.20\(\pm\)1.35 \\  & SpanBERT & 57.54\(\pm\)1.08 & 57.30\(\pm\)0.84 & 52.01\(\pm\)1.72 & 51.48\(\pm\)1.01 & 55.55\(\pm\)1.10 & 55.09\(\pm\)0.74 \\  & JobBERT & 59.81\(\pm\


* [14] M. C. B. K. and R. A. S. G. Wilson, _Theoretical Computer Science_**2006**, 379-485 (2006).
* [15] J. D. L. Miller, "A new approach to the identification of high-order modes in a large number of applications," _Journal of Computational Physics_**199**, no. 1, pp. 1-12, 2000.
* [16] T. W. H. Lee, "A new method for low-order mode decomposition," _IEEE Transactions on Pattern Analysis and Machine Intelligence_**2006**, no. 1, pp. 1-10, 2006.

[MISSING_PAGE_POST]

 

* [14] M. C. R. Jones, J. S. Laird, and D. A. Smith, "Theoretical Population of the Universe: The Case of the New Galaxy," in _Proceedings of the Third International Conference on High Energy Physics_, volume 203 of _Lecture Notes in Computer Science_, vol. 589 of _Lecture Notes in Computer Science_, vol. 67 of _Lecture Notes in Computer Science_, vol. 67 of _Lecture Notes in Computer Science_, vol. 67 of _Lecture Notes in Computer Science_, vol. 68 of _Lecture Notes in Computer Science_, vol. 68 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 70 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 69 of _Lecture Notes in Computer Science_, vol. 71 of _Lecture Notes in Computer Science_, vol. 72 of _Lecture Notes in Computer Science_, vol. 73 of _Lecture Notes in Computer Science_, vol. 74 of _Lecture Notes in Computer Science_, vol. 75 of _Lecture Notes in Computer Science_, vol. 76 of _Lecture Notes in Computer Science_, vol. 77 of _Lecture Notes in Computer Science_, vol. 78 of _Lecture Notes in Computer Science_, vol. 79 of _Lecture Notes in Computer Science_, vol.

 



\begin{table}
\begin{tabular}{l|p{142.3pt} p{85.4pt} p{76.9pt}} \hline  & \multicolumn{3}{c}{**Knowledge**} \\ \hline
**Src.** & **Train** & **Development** & **Test** \\ \hline \multirow{6}{*}{**class**} & english driving license excel cscs card smaths maths ppc spervice design vak/emea policies b bachelor's degree computer science cae setting & full uk driving licence strice planning engdish agile danish business yava aavacript ajavas