Enhancing Job Posting Classification with Multilingual Embeddings and Large Language Models
Authors: Hamit Kavas, Marc Serra-Vidal, Leo Wanner (Pompeu Fabra University, Adevinta Spain, Catalan Institute for Research and Advanced Studies (ICREA), 2024)
Topic: Gender and Inclusive Language Studies

Abstract
In the modern labour market, taxonomies such the European Skills, Competences, Qualifications and Occupations (ESCO) classification are used as an interlingua to match job postings with job seeker profiles. Both are classified with respect to ESCO occupations, and match if they align with the same occupation and the same skills assigned to the occupation. However, matching models usually struggle with the classification because of overlapping skills and similar definitions of occupations defined in the ESCO taxonomy. This often leads to imprecise classification outcomes. In this paper, we focus on the challenge of the classification of job postings written in Italian or Spanish against ESCO occupations written in English. We experiment with multilingual embeddings, zero-shot classification, and use of a large language model (LLM) and show that the use of an LLM leads to best results. Furthermore, we also explore an alternative automatic labeling method by prompting three top-performing LLMs to annotate the test dataset. This approach serves both as an experiment on the usability of automatic labeling and as an evaluation of the reliability of the automatically assigned labels, involving human annotators.

Introduction
The modern labour market becomes more and more diverse. High-tech jobs demand novel skills and competences, which in their turn keep undergoing adaptations and modifications. Under these circumstances, accurately classifying job postings and CVs of job seekers (henceforth candidate experiences) that contain detailed technological specifications with remarkably similar yet distinct skills and experiences has evolved into a complex challenge. The overwhelming majority of job portals and employment agencies use either the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy or its US equivalent O*Net taxonomy} to classify job postings and candidate experiences in terms of job title labeled ESCO/O*Net occupations. Most of the proposals to automatic alignment of job postings with candidate experiences (or vice versa) also use ESCO or O*Net AUTHOR. %but they often do so indirectly by first classifying both the postings and the applications against the occupations and their skills in the chosen taxonomy and then matching the obtained ESCO/O*Net labels. Typically, they also leverage some of the additional information provided in the taxonomy, such as skills, competences, and qualifications -- although, e.g., \citet{yamashita2022james} focus solely on job (i.e., occupation) titles and then incorporate job transition trajectories and historical experiences, including start and end dates, from the job application. However, despite their wide use, both ESCO and O*Net taxonomies exhibit principle limitations for the task of automatic classification of job postings and candidate experiences because due to their tree structure they often fail to adequately distinguish between occupations that exhibit substantial skill overlaps. For instance, two job postings labeled as `data analyst' may appear similar but require different skills if one focuses on market research while the other concentrates on healthcare trends analysis. This issue is particularly pronounced when classification relies on a single label, such as the job title of an ESCO occupation, where skill overlaps undermine precise classification. Hence, employing multiple job titles and framing the problem as a multi-label classification task is imperative. %Although ESCO provides very detailed definitions, any classification method that does not incorporate these comprehensive descriptions, essential skills, and synonyms will miss critical details, rendering effective differentiation between occupations unattainable. This paper addresses the challenge of multilingual multi-label classification using Large Language Models (LLMs) for the alignment of Italian and Spanish job postings with English job titles encountered in the ESCO taxonomy. Multilingual class embeddings are explored to improve classification accuracy, aiming to provide the necessary contextual awareness and addressing the core limitations of taxonomies such as ESCO. %Classifying job postings with respect to ESCO job titles / occupations enables a recommendation system where similarly classified job postings are suggested to recruiters and job seekers. Furthermore, we explore an alternative automatic labeling method by prompting three top-performing LLMs to annotate the test dataset. This approach serves both as an experiment on the usability of automatic labeling and as an evaluation of the reliability of the automatically assigned labels, involving human annotators. To provide LLMs with domain-specific information and to mitigate hallucinations in the course of the classification of the job postings, we employ Retrieval Augmented Generation (RAG) AUTHOR, which combines information retrieval with a generative model. RAG serves two critical functions in our methodology. Firstly, it provides detailed definitions, including essential skills and synonyms for each ESCO occupation, selected through vector similarity as outlined in AUTHOR. Secondly, it ensures that the assigned job titles are restricted to titles within our predefined label space, i.e., standardized job titles defined in the ESCO taxonomy. %In other words, we propose augmenting Large Language Models (LLMs) with ESCO occupations as external knowledge sources, utilizing Retrieval-Augmented Generation (RAG) as an in-context learning approach, specifically through learning by analogy AUTHOR, for job posting classification. Our model incorporates job titles from the ESCO taxonomy as labels and selects an appropriate job title for each job posting in our dataset. \hmt{that was in English and scores were higher there, should we remove the following? The contributions of our work are: \bullet We explore the impact of using multilingual class embeddings derived from the ESCO taxonomy for the task of job posting classification.\\ \bullet We integrate RAG to provide LLMs with domain-specific information and eliminate the dependency on fine-tuning;\\ \bullet We show how the LLM response can be restricted to standardized job titles and thus how LLMs can be used for high quality job title classification that outperforms state-of-the-art proposals for this task. The remainder of the paper is structured as follows. In Section sec:rel-work, we present a concise overview of the related work. In Section sec:model, the model on which our work is based is outlined. Section sec:experiments describes the experiments we carried out, the results we obtained in these experiments, and their discussion. In Section sec:concls, finally, draws some conclusions from the presented work and outlines some directions for future research. In Appendix sec:abl, we present an ablation study in which we assess the comprehension of English ESCO job titles and its Spanish equivalents by our model. Appendix sec:postings provides, for illustration, examples of Italian job postings and predicted ESCO job titles. In Appendix sec:prompts, we present the signature used to prompt Large Language Models for pre-processing.

Conclusion
In this paper, we argued that the use of multilingual embeddings in combination with LLMs significantly enhances our ability to distinguish between very similar (or even identical) job titles that suggest different skills and competencies. Our experiments have shown that this is indeed the case, demonstrating that the combination of multilingual text embeddings similarity with the Llama-3 markedly exceeds the performance of other leading approaches in the field. In the future, we plan to apply the same approach to the analysis and classification of job candidate experiences. Once it is ensured that both job postings and candidate experiences can accurately be modeled using the embedded representation of the ESCO taxonomy, we plan to set the stage for a more direct and efficient alignment process between job postings and experiences of job seekers. Another interesting direction for future research is to analyze the lexical overlap between English domain-specific terms that appear in Italian and Spanish job postings and the English occupation descriptions in the ESCO taxonomy. Such an analysis would reveal whether job types with higher lexical overlap affect model accuracy, providing deeper insights into the multilingual nature of the task. \appendix