ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain
Year: 2023

Abstract:
The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-Rlarge, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-Rlarge on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.

heading: 1 Introduction
text: The dynamic nature of labor markets, driven by technological changes, migration, and digitization, has resulted in a significant amount of job advertisement data (JAD) being made available on various platforms to attract qualified candidates (Brynjolfsson and McAfee, 2011, 2014; Balog et al., 2012). This has led to an increase in tasks related to JAD, including skill extraction (Kivimäki et al., 2013; Zhao et al., 2015; Sayfullina et al., 2018; Smith et al., 2019; Tamburri et al., 2020; Shi et al., 2020; Chernova, 2020; Bhola et al., 2020; Zhang et al., 2022a,b,c; Green et al., 2022; Gnehm et al., 2022;
Beauchemin et al., 2022; Decorte et al., 2022; Goyal et al., 2023), skill classification (Decorte et al., 2022; Zhang et al., 2022b), job title classification (Javed et al., 2015, 2016; Decorte et al., 2021; Green et al., 2022), de-identification of entities in job postings (Jensen et al., 2021), and multilingual skill entity linking (ESCO, 2022).
While some previous studies have focused on JAD in non-English languages (Zhang et al., 2022b; Gnehm et al., 2022; Beauchemin et al., 2022), their baselines have typically relied on language-specific models, either using domain-adaptive pre-training (DAPT; Gururangan et al., 2020) or off-the-shelf models. The lack of comprehensive, open-source JAD data in various languages makes it difficult to fully pre-train a language model (LM) using such data. In this work, we seek external resources that can help improve the multilingual performance on the JAD domain. We use the ESCO taxonomy (le Vrang et al., 2014), which is a standardized system for describing and categorizing the skills, competences, qualifications, and occupations of workers in the European Union. The ESCO taxonomy, which has been curated by humans, covers over 13,000 skills and 3,000 occupations in 27 languages. Therefore, we seek to answer: To what extent can we leverage the ESCO taxonomy to pre-train a domain-specific and language-agnostic model for the computational job market domain?
In this work, we release the first multilingual JAD-related model named ESCOXLM-R, a language model based on XLM-Rlarge that incorporates data from the ESCO taxonomy through the use of two pre-training objectives (Figure 1): Masked Language Modeling (MLM) and a novel ESCO relation prediction task (Section 2). We evaluate ESCOXLM-R on 9 JAD-related datasets in 4 different languages covering 2 NLP tasks (Section 3). Our results show that ESCOXLM-R outperforms previous state-of-the-art (SOTA) on 6 out of 9 datasets (Section 4). In addition, our fine-grained
11871
analysis reveals that ESCOXLM-R performs better on short spans compared to XLM-Rlarge, and consistently outperforms XLM-Rlarge on entity-level and surface-level span-F1 (Section 5).
Contributions In this work, we present and release the following:
• ESCOXLM-R, an XLM-Rlarge-based model, which utilizes domain-adaptive pre-training on the 27 languages from ESCO.1
• The largest JAD evaluation study to date on 3 job-related tasks, comprising 9 datasets in 4 languages and 4 models.
• A fine-grained analysis of ESCOXLM-R’s performance on different span lengths, and emerging entities (i.e., recognition of entities in the long tail).

heading: 2 ESCOXLM-R
text: Preliminaries In the context of pre-training, an LM is trained using a large number of unlabeled documents, X = X(i), and consists of two main functions: fencoder(.), which maps a sequence of tokens X = (x1, x2, ..., xt) to a contextualized vector representation for each token, represented as (h1, h2, ..., ht), and fhead(.), the output layer that takes these representations and performs a specific
1The code for ESCOXLM-R is available as opensource: https://github.com/mainlp/escoxlmr. We further release ESCOXLM-R under an Apache License 2.0 on HuggingFace: https://huggingface.co/jjzha/ esco-xlm-roberta-large.
task, such as pre-training in a self-supervised manner or fine-tuning on a downstream application. For example, BERT (Devlin et al., 2019) is pre-trained using two objectives: MLM and Next Sentence Prediction (NSP). In MLM, a portion of tokens in a sequence X is masked and the model must predict the original tokens from the masked input. In the NSP objective, the model takes in two segments (XA, XB) and predicts whether segment XB follows XA. RoBERTa (Liu et al., 2019) is a variation of BERT that uses dynamic MLM, in which the masking pattern is generated each time a sequence is fed to the LM, and does not use the NSP task.
Multilinguality Both BERT and RoBERTa have been extended to support multiple languages, resulting in multilingual BERT (mBERT; Devlin et al., 2019) and XLM-RoBERTa (XLM-R; Conneau et al., 2020). XLM-R was found to outperform mBERT on many tasks (e.g., Conneau et al., 2020; Hu et al., 2020; Lauscher et al., 2020) due to careful tuning, sampling, and scaling to larger amounts of textual data. Because of this, our ESCOXLM-R model is based on XLM-Rlarge.

heading: 2.1 European Skills, Competences, Qualifications and Occupations Taxonomy
text: The European Skills, Competences, Qualifications, and Occupations (ESCO; le Vrang et al., 2014) taxonomy is a standardized system for describing and categorizing the skills, competences, qualifications, and occupations of workers in the European Union (EU). It is designed to serve as a common lan-
guage for the description of skills and qualifications across the EU, facilitating the mobility of workers by providing a common reference point for the recognition of qualifications and occupations. The taxonomy is developed and maintained by the European Commission and is based on the International Classification of Occupations and the International Standard Classification of Education. It includes 27 European languages: Bulgarian (ar), Czech (cs), Danish (da), German (de), Greek (el), English (en), Spanish (es), Estonian (et), Finnish (fi), French (fr), Gaelic (ga), Croatian (hr), Hungarian (hu), Icelandic (is), Italian (it), Lithuanian (lt), Latvian (lv), Maltese (mt), Dutch (nl), Norwegian (no), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk), Slovenian (sl), Swedish (sv), and Arabic (ar). Currently, it describes 3,008 occupations and 13,890 skills/competences (SKC) in all 27 languages.2
The ESCO taxonomy includes a hierarchical structure with links between occupations, skills, and aliases (OSA). In this work, we focus on the occupation pages and extract the following information from the taxonomy:3
2Note that ESCO now also includes Ukrainian, but this model was trained before that inclusion. We use the ESCO V1.0.9 API to extract the data. ESCO contains an Apache 2.0 and a European Union Public License 1.2.
3An example of the extracted information can be found in Listing 1 (Appendix A), and the original page can be accessed at https://bit.ly/3DY1zsX.
• ESCO Code: The taxonomy code for the specific occupation or SKC.
• Occupation Label: The preferred occupation name (i.e., title of the occupation).
• Occupation Description/Definition: A description of the responsibilities of the specific occupation.
• Major Group Name: The name of the overarching group to which the occupation belongs, e.g., “Veterinarians” for the occupation “animal therapist”.
• Alternative Labels: Aliases for the specific occupation, e.g., “animal rehab therapist” for the occupation “animal therapist”.
• Essential Skills: All necessary SKCs for the occupation, including descriptions of these.
• Optional Skills: All optional SKCs for the occupation, including descriptions of these.
In Figure 2, we present the distribution of pretraining instances and the mean description lengths for each language in the ESCO taxonomy. Note that the number of descriptions is not the same for all languages, and we do not count empty descriptions (i.e., missing translations) for certain occupations or SKCs.

heading: 2.2 Pre-training Setup
text: To improve our XLM-Rlarge-based model, we employ domain-adaptive pre-training techniques as described in previous work such as Alsentzer et al. (2019); Han and Eisenstein (2019); Lee et al. (2020); Gururangan et al. (2020); Nguyen et al. (2020). Given the limited amount of training data (3.72M sentences), we utilize the XLM-Rlarge checkpoint provided by the HuggingFace library (Wolf et al., 2020) as a starting point.4 Our aim is to fine-tune the model to internalize domain-specific knowledge related to occupation and SKCs, while maintaining its general knowledge acquired during the original pre-training phase.
We introduce a novel self-supervised pretraining objective for ESCOXLM-R, inspired by LinkBERT from Yasunaga et al. (2022). We view the ESCO taxonomy as a graph of occupations and SKCs (Figure 1), with links between occupations or occupations and SKCs in various languages. By placing similar occupations or SKCs in the same context window and in different languages, we can learn from the links between (occupation ↔ occupation) and (occupation ↔ SKCs) in different languages for true cross-lingual pre-training. In addition to the MLM pre-training objective, which is used to learn concepts within contexts, we introduce another objective called ESCO Relation Prediction (ERP) to internalize knowledge of connections within the taxonomy in the LM. We take an anchor concept (CA) by concatenating it with its description (XA) from the ESCO taxonomy and sample an additional concept (CB) concatenated with its description (XB) to create LM input [CLS]
4https://huggingface.co/xlm-roberta-large
CAXA [SEP] CBXB [SEP].5 We sample CBXB in three ways with uniform probability:
1. Random: We randomly sample CBXB from the ESCO taxonomy, in any language;
2. Linked: We sample CBXB in any language from the same occupation page, for example, an “animal therapist” (or an alias of the “animal therapist”, e.g., “animal rehab therapist”) should have knowledge of “animal behavior”;
3. Grouped: We sample CBXB from the same major group in any language. For the same example “animal therapist”, it comes from major group 2: Professionals → group 22: Health professionals. Several other concepts, e.g., “Nursing professionals” fall under this major group.
Pre-training Objectives The LM is trained using two objectives. First is the MLM objective, and the second is the ERP objective, where the task is to classify the relation r of the [CLS] token in [CLS] CAXA [SEP] CBXB [SEP] (r ∈ Random,Linked,Grouped). The rationale behind this is to encourage the model to learn the relevance between concepts in the ESCO taxonomy. We formalize the objectives in Equation (1):
L = LMLM + LERP = − ∑
i
log p (xi | hi)− log p (r | h[CLS]) ,
(1) we define the overall loss L as the sum of the MLM loss LMLM and the ERP loss LERP. The MLM loss is calculated as the negative log probability of the input token xi given the representation hi. Similarly, the ERP loss is the negative log probability of the relationship r given the representation of the start-token h[CLS]. In our implementation, we use XLM-Rlarge and classify the start-token [CLS] for ERP to improve the model’s ability to capture the relationships between ESCO occupations and skills.
5The special tokens used in this example follow the naming convention of BERT for readability, [CLS] and [SEP]. However, since we use XLM-Rlarge there are different special tokens: 
 as the beginning of the sequence, 
 as the SEP token, and 

 as segment separators. Formally, given the example in the text: 
 CAXA 

 CBXB 
.
Implementation For optimization we follow (Yasunaga et al., 2022), we use the AdamW (Loshchilov and Hutter, 2019) optimizer with (β1, β2) = (0.9, 0.98). We warm up the learning rate 1e−5 for a ratio of 6% and then linearly decay it. The model is trained for 30K steps, which is equivalent to one epoch over the data, and the training process takes 33 hours on one A100 GPU with tf32. We use a development set comprising 1% of the data for evaluation. In Figure 3, the pre-training loss and performance on the dev. set are plotted, it can be seen that the accuracy plateaus at 30K steps. Though the train and development loss hint that further gains could be obtained on the pretraining objective, we found through empirical analysis on downstream tasks that 30K steps performs best.

heading: 3 Experimental Setup
text: Table 1 provides the details of the downstream datasets used in this study. Most of the datasets are in EN, with a smaller number in DA, DE, and FR. For each dataset, a brief description and the corresponding best-performing models are given. We put examples of each dataset (apart from JobStack due to the license) in Appendix B.
SKILLSPAN (Zhang et al., 2022a) The job posting dataset includes annotations for skills and knowledge, derived from the ESCO taxonomy. The best model in the relevant paper, JobBERT, was retrained using a DAPT approach on a dataset of 3.2 million EN job posting sentences. This is the bestperforming model which we will compare against.
KOMPETENCER (Zhang et al., 2022b) This dataset is used to evaluate models on the task
of classifying skills according to their ESCO taxonomy code. It includes EN and DA splits, with the EN set derived from SKILLSPAN. There are three experimental setups for evaluation: fully supervised with EN data, zero-shot classification (EN→DA), and few-shot classification (a few DA instances). The best-performing model in this work is RemBERT (Chung et al., 2021), which obtains the highest weighted macro-F1 for both EN and DA. In this work, we use setup 1 and 3, where all available data is used.
BHOLA (Bhola et al., 2020) The task of this EN job posting dataset is multilabel classification: Predicting a list of necessary skills in for a given job description. It was collected from a Singaporean government website. It includes job requirements and responsibilities as data fields. Pre-processing steps included lowercasing, stopword removal, and rare word removal. Their model is BERT with a bottleneck layer (Liu et al., 2017). In our work, the bottleneck layer is not used and no additional training data is generated through bootstrapping. To keep comparison fair, we re-train their model without the additional layer and bootstrapping. We use Mean Reciprocal Rank as the main results metric.
SAYFULLINA (Sayfullina et al., 2018) This dataset is used for soft skill prediction, a sequence labeling problem. Soft skills are personal qualities that contribute to success, such as “team working”, “being dynamic”, and “independent”. The models for this dataset include a CNN (Kim, 2014), an LSTM (Hochreiter et al., 1997), and a Hierarchical Attention Network (Yang et al., 2016). We compare to their best-performing LSTM model.
GREEN (Green et al., 2022) A sentence-level sequence labeling task involving labeling skills, qualifications, job domain, experience, and occupation labels. The job positions in the dataset are from the United Kingdom. The industries represented in the data vary and include IT, finance, healthcare, and sales. Their model for this task is a Conditional Random Field (Lafferty et al., 2001) model.
JOBSTACK (Jensen et al., 2021) This corpus is used for de-identifying personal data in job vacancies on Stack Overflow. The task involves sequence labeling and predicting Organization, Location, Name, Profession, and Contact details labels. The best-performing model for this task is a transformer-based (Vaswani et al., 2017) model trained in a multi-task learning setting. Jensen et al. (2021) propose to use the I2B2/UTHealth corpus, which is a medical de-identification task (Stubbs and Uzuner, 2015), as auxiliary data, which showed improvement over their baselines.
GNEHM (Gnehm et al., 2022) A Swiss-German job ad dataset where the task is Information and Communications Technology (ICT)-related entity recognition, these could be ICT tasks, technology stack, responsibilities, and so forth. The used dataset is a combination of two other Swiss datasets namely the Swiss Job Market Monitor and an online job ad dataset (Gnehm and Clematide, 2020; Buchmann et al., 2022). Their model is dubbed JobGBERT and is based on DAPT with German BERTbase (Chan et al., 2020).
FIJO (Beauchemin et al., 2022) A French job ad dataset with the task of labeling skill types using a sequence labeling approach. The skill groups are based on the AQESSS public skills repositories
and proprietary skill sets provided by their collaborators. These skill types are divided into four categories: “Thoughts”, “Results”, “Relational”, and “Personal”. The best-performing model for this task is CamemBERT (Martin et al., 2020).

heading: 4 Results
text: The results of the models are presented in Table 2. To evaluate the performance, four different models are used in total: ESCOXLM-R, the best-performing model originally reported in the relevant paper for the downstream task, vanilla XLM-Rlarge, and an XLM-Rlarge model that we continuously pre-trained using only MLM (DAPT; excluding the ERP objective) using the same pre-training hyperparameters as ESCOXLM-R. For more information regarding the hyperparameters of fine-tuning, we refer to Appendix C (Table 5).
English ESCOXLM-R is the best-performing model in 4 out of 6 EN datasets. The largest improvement compared to the previous SOTA is observed in SAYFULLINA and GREEN, with over 19 F1 points. In 3 out of 4 datasets, ESCOXLM-R has the overall lower standard deviation. For JOBSTACK, the previous SOTA performs best, and for KOMPETENCER, XLM-Rlarge (+ DAPT) has the highest performance.
Non-English In 2 out of 3 datasets, ESCOXLM-R improves over the previous SOTA, with the largest absolute difference on French FIJO with 10.3 F1 points. In the Danish subset of KOMPETENCER, XLM-Rlarge (+ DAPT) has higher performance than ESCOXLM-R. Next, we will discuss potential reasons for these differences.

heading: 4.1 Analysis
text: We highlight that the performance gains of ESCOXLM-R are generally much larger than any of the losses, indicating a largely positive effect of training on ESCO. The improved performance of ESCOXLM-R on JAD datasets in Table 2 is likely due to the focus on tasks with token-level annotation (i.e., sequence labeling). This suggests that pretraining on the ESCO taxonomy is particularly useful for these types of tasks. The under-performance of ESCOXLM-R on the KOMPETENCER dataset in both EN and DA may be because the task involves predicting the ESCO taxonomy code for a given skill without context, where we expect ESCO to particularly help with tasks where having context is relevant. We suspect applying DAPT and ERP on ESCO specifically improves recognizing entities that are uncommon. On the other hand, the poor performance on the JOBSTACK dataset may be due to the task of predicting various named entities,
such as organizations and locations. By manual inspection, we found that ESCO does not contain entities related to organizations, locations, or persons, thus this reveals that there is a lack of relevant pre-training information to JOBSTACK.

heading: 5 Discussion

heading: 5.1 Performance on Span Length
text: We seek to determine whether the difference in performance between the ESCOXLM-R and XLM-Rlarge models is due to shorter spans, and to what extent. One application of predicting short spans well is the rise of technologies, for which the names are usually short in length. Zhang et al. (2022c) observes that skills described in the ESCO dataset are typically short, with a median length of approximately 3 tokens. We compare the average performance of both models on the test sets of each dataset, where span-F1 is used as measurement. We group gold spans into buckets of lengths 1-2, 3-4, 5-6, 7-8,
and 9-10, and present the span-F1 for each model (XLM-Rlarge vs. ESCOXLM-R) in each bucket.
Shown in Figure 4, ESCOXLM-R outperforms XLM-Rlarge on shorter spans (i.e., 1-2 or 3-4) in 6 out of the 6 datasets, suggesting that pre-training on ESCO is beneficial for predicting short spans. However, there is a slight decline in performance on some datasets (e.g., SKILLSPAN, JOBSTACK, and GNEHM) when the spans are longer (i.e., 7-8 or 9-10). It is worth noting that the number of instances in these longer span buckets is lower, and therefore errors may be less apparent in terms of their impact on overall performance.

heading: 5.2 Entity-F1 vs. Surface-F1
text: In this analysis, we adopt the evaluation method used in the W-NUT shared task on Novel and Emerging Entity Recognition (Derczynski et al., 2017). In this shared task, systems are evaluated using two measures: entity span-F1 and surface span-F1. Entity span-F1 assesses the precision, recall, and harmonic mean (F1) of the systems at the entity level, while surface span-F1 assesses their ability to correctly recognize a diverse range of entities, rather than just the most frequent surface forms. This means surface span-F1 counts entity types, in contrast to entity tokens in the standard entity span-F1 metric.
As shown in Table 3, we first calculate the ratio of unique entities and total entities in each relevant train set (i.e., datasets where we do span labeling). A higher ratio number indicates a wider variety of spans. Both XLM-Rlarge and ESCOXLM-R tend to have lower performance when variety gets high (above 0.75). In addition, there are 2 datasets (SAYFULLINA, JOBSTACK) where we see a low
variety of spans and large discrepancy between performance of entity span-F1 and surface span-F1. This difference is lower for ESCOXLM-R (especially in SAYFULLINA) suggesting that pre-training on ESCO helps predicting uncommon entities.
It is also noteworthy that the standard deviations for the scores at the entity span-F1 are generally lower than those for the surface span-F1. This suggests that the results for the entity span-F1 scores are more consistent across different runs, likely due to recognizing common entities more.
Overall, ESCOXLM-R consistently outperforms XLM-Rlarge in both the entity-level and surface-level F1 scores, indicating the benefits of using the ESCO dataset for pre-training on JAD tasks.

heading: 6 Related Work
text: To the best of our knowledge, we are the first to internalize an LM with ESCO for job-related NLP tasks. There are, however, several works that integrate factual knowledge (i.e., knowledge graphs/bases) into an LM. Peters et al. (2019) integrates multiple knowledge bases into LMs to enhance their representations with structured, human-curated knowledge and improve perplexity, fact recall and downstream performance on various tasks. Zhang et al. (2019); He et al. (2020); Wang et al. (2021b) combine LM training with knowledge graph embeddings. Wang et al. (2021a) introduces K-Adapter for injecting knowledge into pre-trained models that adds neural adapters for each kind of knowledge domain. Yu et al. (2022) introduces Dict-BERT, which incorporates definitions of rare or infrequent words into the input sequence and further pre-trains a BERT model.
Calixto et al. (2021) introduced a multilingual Wikipedia hyperlink prediction intermediate task to improve language model pre-training. Similarly, Yasunaga et al. (2022) introduced LinkBERT which leverages links between documents, such as hyperlinks, to capture dependencies and knowledge that span across documents by placing linked documents in the same context and pre-training the LM with MLM and document relation prediction.

heading: 7 Conclusion
text: In this study, we introduce ESCOXLM-R as a multilingual, domain-adapted LM that has been further pre-trained on the ESCO taxonomy. We evaluated ESCOXLM-R, to the best of our knowledge, on the broadest evaluation set in this domain on 4 different languages. The results showed that ESCOXLM-R outperformed XLM-Rlarge on job-related downstream tasks in 6 out of 9 datasets, particularly when the task was relevant to the ESCO taxonomy and context was important. It was found that the improvement of ESCOXLM-R was mainly due to its performance on shorter span lengths, demonstrating the value of pre-training on the ESCO dataset. ESCOXLM-R also demonstrated improved performance on both frequent surface spans and a wider range of spans. Overall, this work showed the potential of ESCOXLM-R as an LM for multilingual job-related tasks. We hope that it will encourage further research in this area.
Limitations
There are several limitations to this study that should be considered. First, a key limitation is the lack of a variety of language-specific JAD. Here, we have four different languages namely EN, DA, FR, and DE. This means that our analysis is based on a limited subset of languages and may not be representative of JAD data outside of these four languages.
In turn, the second limitation is that the ESCO taxonomy used as pre-training data only covers Europe and the datasets used in this work also covers mostly Europe. The results may not be generalizable to other regions. However, we see a slight improvement in the BHOLA dataset, the data of which comes from Singapore, which hints that it could generalize to other cultures.
The ESCO relation prediction task aims for learning the relations between elements of the ESCO taxonomy. We acknowledge that we do
not evaluate the effectiveness of the pre-training objective in relation-centered tasks. Unfortunately, to the best of our knowledge, there is no job-related dataset containing relations between skill/occupation concepts to benchmark our model on. We consider this interesting future work.
Finally, we did not conduct an ablation study on the ERP pre-training objective, i.e., which errors it makes. As the accuracy of the objective is 60%, we are unable to determine which sampling method is detrimental to this accuracy. However, we suspect that the Linked sampling approach might be the hardest to predict correctly. For example, many occupations have a lot of necessary and optional skills, thus it is harder to determine if some skill truly belongs to a specific occupation. Nevertheless, we see that adding the ERP objective improves over regular MLM domain-adaptive pre-training.
Despite these limitations, we believe that this study provides valuable resources and insights into the use of ESCOXLM-R for analyzing JAD and suggests directions for future research. Future studies could address the limitations of this study by using a larger, more diverse datasets and by conducting ablation studies on the language model to better understand which parts contribute to the results.
Ethics Statement
We also see a potential lack of language inclusiveness within our work, as we addressed in the Limitation section that ESCO mostly covers Europe (and the Arabic language). Nevertheless, we see ESCOXLM-R as a step towards inclusiveness, due to JAD frequently being English-only. In addition, to the best of our knowledge, ESCO itself is devoid of any gendered language, specifically, pronouns and other gender-specific terms in, e.g., occupations. However, we acknowledge that LMs such as ESCOXLM-R could potentially be exploited in the process of hiring candidates for a specific job with unintended consequences (unconscious bias and dual use). There exists active research on fairer recommender systems (e.g., bias mitigation) for human resources (e.g., Mujtaba and Mahapatra, 2019; Raghavan et al., 2020; Deshpande et al., 2020; Köchling and Wehner, 2020; SánchezMonedero et al., 2020; Wilson et al., 2021; van Els et al., 2022; Arafan et al., 2022).

heading: Acknowledgements
text: We thank both the NLPnorth and MaiNLP group for feedback on an earlier version of this paper. This research is supported by the Independent Research Fund Denmark (DFF) grant 9131-00019B and in parts by ERC Consolidator Grant DIALECT 101043235.

heading: 2 "id": int,

heading: 3 "esco_code": "2250.4",

heading: 4 "preferred_label": "animal therapist",

heading: 5 "major_group": {

heading: 6 "title": "Veterinarians",
text: 7 "description": "Veterinarians diagnose, [...]" 8 },

heading: 9 "alternative_label": [
text: 10 "animal convalescence therapist", 11 "animal rehab therapist", 12 "animal rehabilitation therapist", 13 "animal therapists", 14 "animal therapist" 15 ], 16 "description": "Animal therapists provide [...]", 17 "essential_skills": [ 18 { 19 "title": "anatomy of animals", 20 "description": "The study of animal body parts, [...]" 21 }, 22 ... 23 ], 24 "optional_skills": [ 25 { 26 "title": "use physiotherapy for treatment of animals", 27 "description": "Adapt human physical therapy [...]" 28 }, 29 ... 30 ] 31
32 }
Listing 1: Example Extraction. An example of the information that is given for ESCO code 2250.4: animal therapist. The original page can be found here: http://data.europa.eu/esco/occupation/ 0b2d3242-22a3-4de5-bd29-efd39cdf2c31.

heading: 1 Experience O O
text: 2 in O O

heading: 3 working B-Skill O

heading: 4 on I-Skill O

heading: 5 a I-Skill O

heading: 6 cloud-based I-Skill O

heading: 7 application I-Skill O

heading: 8 running O O
text: 9 on O O
10 Docker O B-Knowledge 11 . O O 12
13 A O O 14 degree O B-Knowledge 15 in O I-Knowledge 16 Computer O I-Knowledge 17 Science O I-Knowledge 18 or O O 19 related O O 20 fields O O 21 . O O
Listing 2: Data Example SkillSpan.

heading: B Data Examples
text: SKILLSPAN Listing 2 SAYFULLINA Listing 3 GREEN Listing 4 BHOLA Listing 5 KOMPETENCER Listing 6 FIJO Listing 7 GNEHM Listing 8
Table 4: Data example references for each dataset.
1 ability O 2 to O

heading: 3 work B-Skill

heading: 4 under I-Skill

heading: 5 stress I-Skill
text: 6 condition O 7
8 due O 9 to O
10 the O 11 dynamic B-Skill 12 nature O 13 of O 14 the O 15 group O 16 environment O 17 , O 18 the O 19 ideal O 20 candidate O 21 will O
Listing 3: Data Example Sayfullina.
1 A O

heading: 2 sound O
text: 3 understanding O 4 of O

heading: 5 the O

heading: 6 Care B-Skill

heading: 7 Standards I-Skill

heading: 8 together O
text: 9 with O
10 a O 11 Nursing B-Qualification 12 qualification I-Qualification 13 and O 14 current O 15 NMC B-Qualification 16 registration I-Qualification 17 are O 18 essential O 19 for O 20 this O 21 role O 22 . O
Listing 4: Data Example Green.

heading: 1 department economics national university singapore invites applications

heading: 2 teaching oriented positions level lecturer senior lecturer [...] <labels>
text: Listing 5: Data Example Bhola.

heading: 1 <English>

heading: 2 team worker S4

heading: 3 passion for developing your career S1

heading: 4 liaise with internal teams S1

heading: 5 identify system requirements S2
text: 6 plan out our new features S4 7

heading: 8 <Danish>

heading: 9 arbejde med børn i alderen ½-3 år S3
text: 10 samarbejde S1 11 fokusere på god kommunikation S1 12 bidrage til at styrke fællesskabet S1 13 ansvarsbevidst A1 14 lyst til et aktivt udeliv A1
Listing 6: Data Example Kompetencer.

heading: 1 Participer B-relationnel

heading: 2 au I-relationnel

heading: 3 réseau I-relationnel

heading: 4 téléphonique I-relationnel

heading: 5 mis O

heading: 6 sur O

heading: 7 pied O

heading: 8 lors O

heading: 9 des O
text: 10 campagnes O 11 d'inscription O 12 pour O 13 fournir B-pensee 14 les I-pensee 15 renseignements I-pensee 16 nécessaires I-pensee 17 aux I-pensee 18 assurés I-pensee
Listing 7: Data Example Fijo.
1 in O

heading: 2 mit O

heading: 3 guten O
text: 4 EDV-Kenntnissen B-ICT 5
6 . O 7 Es O

heading: 8 erwartet O

heading: 9 Sie O
text: 10 eine O 11 interessante O 12 Aufgabe O 13 in O 14 einer O 15 Adressverwaltung O 16 ( O 17 Rechenzenter B-ICT 18 ) O
Listing 8: Data Example Gnehm.

heading: C Fine-tuning Details
text: For fine-tuning XLM-Rlarge (+ DAPT) and ESCOXLM-R on the downstream tasks, we use MaChAmp (van der Goot et al., 2021). For more details we refer to their paper. We always include the original learning rate, batch size, maximum sequence length, and epochs from the respective downstream tasks in our search space (whenever applicable). Each model is trained on an NVIDIA A100 GPU with 40GBs of VRAM and an AMD Epyc 7662 CPU. The seed numbers the models are initialized with are 276800, 381552, 497646, 624189, 884832. We run all models with the maximun number of epochs indicated in Table 5 and select the best-performing one based on validation set performance in the downstream metric.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly
B 3 Did you use or create scientific artifacts? Section 1, Section 2, Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, 2, 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Footnote 1+2, Table 1, we plan to release the model under an Apache 2.0 License.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Partially reflected in the Ethics Statement

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The JobStack dataset contains some privacy-bearing entities. We re-train the models on the anonymized dataset.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Table 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1
C 3 Did you run computational experiments? Section 2, 3, 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 2, and 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2.2, and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We consider this interesting future work. Finally, we did not conduct an ablation study on the ERP pre-training objective, i. e. , which errors it makes. As the accuracy of the objective is 60%, we are unable to determine which sampling method is detrimental to this accuracy. However, we suspect that the Linked sampling approach might be the hardest to predict correctly. For example, many occupations have a lot of necessary and optional skills, thus it is harder to determine if some skill truly belongs to a specific occupation. Nevertheless, we see that adding the ERP objective improves over regular MLM domain-adaptive pre-training. Despite these limitations, we believe that this study provides valuable resources and insights into the use of ESCOXLM-R for analyzing JAD and suggests directions for future research. Future studies could address the limitations of this study by using a larger, more diverse datasets and by conducting ablation studies on the language model to better understand which parts contribute to the results