SlowGuess/NAACL_2022_OCR huggingface dataset:
Setup The data is structured as CoNLL format (Tjong Kim Sang, 2002). For the nested annotations, the skill tags are appearing only in the first column and the knowledge tags are only appearing in the second column of the file and they are allowed to overlap with each other. We perform experiments with single-task learning (STL) on either the skill or knowledge components, MTL for predicting both skill and knowledge tags at the same time, while evaluating the MTL models also on either skills or knowledge components. We used a single joint MTL model with hard-parameter sharing (Caruana, 1997). All models are with a final Conditional Random Field (CRF; Lafferty et al.,
7https://huggingface.co/jjzha/ jobbert-base-cased 8https://huggingface.co/jjzha/ jobspanbert-base-cased
Figure 4: Almost Stochastic Order Scores of the Test Set. ASO scores expressed in  $\epsilon_{\mathrm{min}}$ . The significance level  $\alpha = 0.05$  is adjusted accordingly by using the Bonferroni correction (Bonferroni, 1936). Read from row to column: E.g., in COMBINED STL-JobBERT (row) is stochastically dominant over STL-BERT<sub>base</sub> (column) with  $\epsilon_{\mathrm{min}}$  of 0.00.
2001) layer. Earlier research, such as Souza et al. (2019); Jensen et al. (2021) show that BERT models with a CRF-layer improve or perform similarly to its simpler variants when comparing the overall F1 and make no tagging errors (e.g., B-tag follows I-tag). In the case of MTL we use one for each tag type (skill and knowledge). In the STL experiments we use one CRF for the given tag type.
We use the MACHAMP toolkit (van der Goot et al., 2021) for our experiments. For each setup we do five runs (i.e., five random seeds). For evaluation we use span-level precision, recall, and F1, where the F1 for the MTL setting is calculated as described in Benikova et al. (2014).
The results of the experiments are given in Figure 3. We show the average performance of each model in F1 and respective standard deviation over the development and test split. Exact scores on each source split and other metric details are provided in Appendix E. As mentioned before, we experiment with the following settings: SKILL, we train and predict only on skills. KNOWLEDGE, train and only predict for knowledge. COMBINED, we merge the STL predictions of both skills and knowledge. We also train the models in an MTL setting, predicting both skills and knowledge simultaneously. We evaluate the MTL model on both SKILL and KNOWLEDGE separately, and also compare it against the aggregated STL predictions.
Performance on Development Set In Figure 3, we show the results on the development set in the upper plot. We observe similar performance between the domain-adapted STL models—JobBERT and JobSpanBERT—have similar span-F1 for
SKILL:  $60.05 \pm 0.70$  vs.  $60.07 \pm 0.70$ . In contrast, for KNOWLEDGE, BERT<sub>base</sub> and JobBERT are closest in predictive performance:  $60.44 \pm 0.58$  vs.  $60.66 \pm 0.43$ . In the COMBINED setting, JobBERT performs highest with a span-F1 of  $60.32 \pm 0.39$ . On average, JobBERT performs best over all three settings. Surprisingly, the models for both SKILL and KNOWLEDGE perform similarly (around 60 span-F1), despite the sources' differences in properties and length Figure 2. In addition, we find that MTL is not performing better than STL across sources. For exact numbers and source-level (i.e., BIG, HOUSE, TECH), we refer to Appendix E.
Performance on Test Set We select the best performing models in the development set evaluation and apply it to the test set. Results are in Figure 3 in the bottom plot. Since JobBERT and JobSpanBERT are performing similarly, we apply both to the test set and  $\mathrm{BERT}_{\mathrm{base}}$ . We observe a deviation from the development set to the test set: JobSpanBERT  $60.07 \pm 0.30 \rightarrow 56.64 \pm 0.83$  on SKILL, JobBERT  $60.66 \pm 0.43 \rightarrow 63.88 \pm 0.28$  on KNOWLEDGE. For COMBINED, JobBERT performs slightly worse:  $60.32 \pm 0.39 \rightarrow 59.73 \pm 0.38$ . Similar to the development set, we find that on all three methods of evaluation (i.e., SKILL, KNOWLEDGE, and COMBINED), STL still outperforms MTL. For SKILL and KNOWLEDGE, STL is almost stochastically dominant over MTL (i.e., significant), and for COMBINED there is stochastic dominance of STL over MTL, indicated in the next paragraph.
Significance We compare all pairs of models based on five random seeds each using Almost Stochastic Order (ASO; Dror et al., 2019) tests with a confidence level of  $\alpha = 0.05$ . The ASO scores of the test set are indicated in Figure 4. We show that MTL-JobSpanBERT for SKILL shows
For reproducibility, we refer to Appendix D.
almost stochastic dominance  $(\epsilon_{\mathrm{min}} < 0.5)$  over all other models. For KNOWLEDGE and COMBINED, We show that STL-JobBERT is stochastically dominant  $(\epsilon_{\mathrm{min}} = 0.0)$  over all the other models. For more details, we refer to Appendix F for ASO scores on the development set.
What Did Not Work Additionally, we experiment whether representing the entire JP for extracting tokens yields better results than the experiments so far, which were sentence-by-sentence processing setups. To handle entire JPs and hence much longer sequences we use a pre-trained Longformer<sub>base</sub> (Beltagy et al., 2020) model. The document length we use in the experiments is 4096 tokens. Results of the Longformer on the test set are lower: For skills, JobSpanBERT against Longformer results in  $56.64 \pm 0.83$  vs.  $52.55 \pm 2.39$ . For KNOWLEDGE, JobBERT against Longformer shows  $63.88 \pm 0.28$  vs.  $57.26 \pm 1.05$ . Last, for COMBINED, JobBERT against Longformer results in  $59.73 \pm 0.38$  vs.  $55.05 \pm 0.71$ . This drop in performance is difficult to attribute to a concrete reason: e.g., the Longformer is trained on more varied sources than BERT, but not specifically for JPs, which may have contributed to this gap. Since the vanilla Longformer already performs worse than  $\mathrm{BERT}_{\mathrm{base}}$  overall, we did not opt to apply domain-adaptive pre-training. Overall, we show that representing the full JP is not beneficial for SE, at least not in the Longformer setup tested here.
Continuous Pretraining helps SE As previously mentioned, due to the domain specialization of the domain-adapted pre-trained BERT models, they predict more skills and frequently perform better in terms of precision, recall, and F1 as compared to their non-adaptive counterparts. This is especially encouraging as we confirm findings that continuous pre-training helps to adapt models to a specific domain (Alsentzer et al., 2019; Lee et al., 2020; Gururangan et al., 2020; Nguyen et al., 2020). However, there are exceptions. Particularly in Table 5 on TEST for KNOWLEDGE, BERT<sub>base</sub> comes closer in predictive performance to JobBERT (difference of 1.5 F1) than on SKILLS. Our intuition is that knowledge components are often already in the pre-training data (e.g., Wikipedia pages of certain competences like Python, Java etc.) and therefore adaptive pre-training does not substantially boost performance.
Figure 5: Average Length of Predictions of Single Models. We show the average length of the predictions versus the length of our annotated skills and knowledge components on the test set and the total number of predicted skills and knowledge tags in each respective split (#). There is a consistent trend over the three sources.
Difference in Length of Predictions The main motivation of selecting models optimized for long spans was the length of the annotations (Figure 2). We investigate the average length of predictions of each model (Figure 5) to find out whether the models that are adapted to handle longer sequences truly predict longer spans. Interestingly, the average length of predicted skills are longer than the annotations over all three sources. There is a consistent trend among SKILL: BIG and TECH have similar length over predictions  $(>4)$ , while HOUSE is usually lower than length 3. For both BIG and TECH, JobSpanBERT predicts the longest skill spans (4.51 and 4.48 respectively). We suspect due to the domain-adaptive pre-training on JPs, it improved the span prediction performance. In contrast, the Longformer predicts shorter spans. Note that the Longformer is not domain-adapted to JPs.
Regarding KNOWLEDGE, there is also a consistent trend: BIG has the overall longest prediction length while TECH has the lowest. The Longformer predicts the longest spans on average for BIG and TECH. Knowledge components are representative of a normal-length NER task and might not need a specialized model for long sequences. We show the exact numbers in Table 7 (Appendix E) and the number of predicted SKILL and KNOWLEDGE: JobBERT and JobSpanBERT have higher recall than the other models.
Performance per Span Length SKILLS are generally longer than KNOWLEDGE components in our dataset (Figure 2). The previous overall results on the test set (Figure 3) show that performance on
F1-score per Span Length on Test (JobBERT)
Figure 6: Average Span-F1 per Span Length. We bucket the performance of JobBERT according to the length of the spans until 10 tokens and show the performance on each length, averaged over five random seeds. Indicated per bar is the support. The model performs best on medium-length skill spans (i.e., spans with token length of 4-5). For knowledge spans, on average, it performs best on short-length spans (i.e., spans with token length of 1-2).
SKILL is substantially lower than KNOWLEDGE. We therefore investigate whether this performance difference is attributed to the longer spans in SKILL. In Figure 6, we show the average performance of the best performing model (JobBERT) on the three sources (test set) based on the gold span length, until a length of 10.
In SKILL components (upper plot), we see much support for spans with length 1 and 2, which then lowers once the spans become longer. Spans with length of 1 shows low performance on BIG and TECH (around 40 span-F1), which influences the total span-F1. Short skills are usually soft skills, such as "passionate", which can be used as a skill or not. This might confuse the model. In contrast, performance effectively stays similar (around 60 span-F1) for span length of 2 till 7 for all sources. Afterwards, it drops in performance. Thus, the weak performance on SKILL seem to be due to lower performance on the short spans.
For the KNOWLEDGE components (lower plot), they are generally shorter. We see that there is a gap in support between the sources, TECH has a larger number of gold labels compared to BIG and HOUSE. Unlike soft skills, KCs usually consist of proper nouns such as "Python", "Java", and so forth, which connects to the high performance on TECH (around 76 span-F1). Furthermore, support for spans longer than 2 drops considerably. In this
case, if the model predicts a couple of instances correctly, it would substantially increase span-F1. Contrary to SKILL, high performance of KNOWLEDGE can be attributed to its strong performance on short spans.
We present a novel dataset for skill extraction on English job postings—SKILLSpan—and domain-adapted BERT models—JobBERT and JobSpanBERT. We outline the dataset and annotation guidelines, created for hard and soft skills annotation on the span-level. Our analysis shows that domain-adaptive pre-training helps to improve performance on the task for both skills and knowledge components. Our domain-adapted JobSpanBERT performs best on skills and JobBERT on knowledge. Both models achieve almost stochastic dominance over all other models for skills and knowledge extraction, whereas JobBERT in the STL setting achieves stochastic dominance over other models.
With the rapid emergence of new competences, our new approach of skill extraction has future potential, e.g., to enrich knowledge bases such as ESCO with unseen skills or knowledge components, and in general, contribute to providing insights into labor market dynamics. We hope our dataset encourages research into this emerging area of computational job market analysis.
We thank Google's TFRC for their support in providing TPUs for this research. Furthermore, we thank the NLPnorth group for feedback on an earlier version of this paper—in particular, Elisa Bassignana and Max Müller-Eberstein for insightful discussions. We would also like to thank the anonymous reviewers for their comments to improve this paper. Last, we also thank NVIDIA and the ITU High-performance Computing cluster for computing resources. This research is supported by the Independent Research Fund Denmark (DFF) grant 9131-00019B.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Krisztian Balog, Yi Fang, Maarten De Rijke, Pavel Serdyukov, and Luo Si. 2012. Expertise retrieval. Foundations and Trends in Information Retrieval, 6(2-3):127-256.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. *ArXiv* preprint, abs/2004.05150.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Darina Benikova, Chris Biemann, and Marc Reznicek. 2014. NoSta-D named entity annotation for German: Guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2524-2531, Reykjavik, Iceland. European Language Resources Association (ELRA).
Akshay Bhola, Kishaloy Halder, Animesh Prasad, and Min-Yen Kan. 2020. Retrieving skills from job descriptions: A language model based extreme multi-label classification framework. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5832-5842, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Carlo Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilita. *Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze*, 8:3-62.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Mariia Chernova. 2020. Occupational skills extraction with FinBERT. Master's Thesis.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance - how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2773-2785, Florence, Italy. Association for Computational Linguistics.
Peter Elias. 1997. Occupational classification (isco-88): Concepts, methods, reliability, validity and cross-national comparability. Technical report, OECD Publishing.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619.
Akshay Gugnani and Hemant Misra. 2020. Implicit skills extraction using document embedding and its use in job recommendation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 13286-13293. AAAI Press.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238-4248, Hong Kong, China. Association for Computational Linguistics.
Sepp Hochreiter, Jürgen Schmidhuber, and Corso Elvezia. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Faizan Javed, Phuong Hoang, Thomas Mahoney, and Matt McNair. 2017. Large-scale occupational skills normalization for online recruitment. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4627-4634. AAAI Press.
Kristian Nørgaard Jensen, Mike Zhang, and Barbara Plank. 2021. De-identification of privacy-related entities in job postings. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 210–221, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Shanshan Jia, Xiaoan Liu, Ping Zhao, Chang Liu, Lianying Sun, and Tao Peng. 2018. Representation of jobskill in artificial intelligence with knowledge graph analysis. In 2018 IEEE Symposium on Product Compliance Engineering-Asia (ISPCE-CN), pages 1-6. IEEE.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Imane Khaouja, Ismail Kassou, and Mounir Ghogho. 2021. A survey on skill identification from online job ads. IEEE Access, 9:118134-118153.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ilkka Kivimäki, Alexander Panchenko, Adrien Dessy, Dries Verdegem, Pascal Francq, Hugues Bersini, and Marco Saerens. 2013. A graph-based approach to skill extraction from text. In Proceedings of TextGraphs-8 Graph-based Methods for Natural Language Processing, pages 79-87, Seattle, Washington, USA. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282-289. Morgan Kaufmann.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159-174.
Martin le Vrang, Agis Papantoniou, Erika Pauwels, Pieter Fannes, Dominique Vandensteen, and Johan De Smedt. 2014. Esco: Boosting job matching in
europe with semantic interoperability. Computer, 47(10):57-64.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Shan Li, Baoxu Shi, Jaewon Yang, Ji Yan, Shuai Wang, Fei Chen, and Qi He. 2020. Deep job understanding at linkedin. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 2145-2148. ACM.
Liting Liu, Wenzheng Zhang, Jie Liu, Wenxuan Shi, and Yalou Huang. 2021. Learning multi-graph neural network for data-driven job skill prediction. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/doccano.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, Online. Association for Computational Linguistics.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3066-3079, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. ArXiv preprint, abs/1803.09578.
Luiza Sayfullina, Eric Malmi, and Juho Kannala. 2018. Learning representations for soft skill matching. In International Conference on Analysis of Images, Social Networks and Texts, pages 141-152.
Baoxu Shi, Jaewon Yang, Feng Guo, and Qi He. 2020. Salience and market-aware skill extraction for job targeting. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 2871-2879. ACM.