Learning Job Title Representation from Job Description Aggregation Network (2024)

Abstract: Learning job title representation is a vital process for developing automatic human resource tools. To do so, existing methods primarily rely on learning the title representation through skills extracted from the job description, neglecting the rich and diverse content within. Thus, we propose an alternative framework for learning job titles through their respective job description (JD) and utilize a Job Description Aggregator component to handle the lengthy description and bidirectional contrastive loss to account for the bidirectional relationship between the job title and its description. We evaluated the performance of our method on both in-domain and out-of-domain settings, achieving a superior performance over the skill-based approach.

heading: 1 Introduction
text: With the rapid expansion of online recruitment platforms, vast amounts of job advertisement data (JAD) have been generated. One key part of this data is a job posting, providing detailed information on job titles, specialties, and responsibilities for open positions. Thus, the availability of a system that could understand the post semantics, especially job titles, would greatly facilitate the matchmaking process between both the recruiters and job applicants. This leads to a surge of interest in learning job title representation due to its potential ability to automate job-related tasks such as job recommendation (Kaya and Bogers, 2021; Zhao et al., 2021), job trajectory prediction (Decorte et al., 2023a), and job title benchmarking (Zhang et al., 2019).
To learn the title representation, previous works have primarily relied on utilizing skills information to learn the association between the job title and their respective skill (Decorte et al., 2021; Zbib et al., 2022; Bocharova et al., 2023). However, this approach also has some shortcomings as it requires skill information. The skills for a given job are
either manually listed, which can be erroneous or incomplete, or automatically extracted from the job description through methods such as keyword matching or automatic skill extraction (Zhang et al., 2022b; Li et al., 2023). These skill extraction methods often require a predefined skill vocabulary or a curated dataset (Zhang et al., 2022a). Furthermore, it is necessary to keep these resources up-to-date with trends in the job market as the dynamic and rapid growth of emerging job roles.
Previous works mitigate these problems by generating synthetic skill data (Decorte et al., 2023b; Clavié and Soulié, 2023) or creating datasets where both job titles, and skill lists are readily available (Bhola et al., 2020; Goyal et al., 2023). Nonetheless, the former approach further increases pipeline complexity, while the latter suffers from missing skills annotation caused by a communication gap between employers and recruiters.
In this work, we propose to overcome the challenges of obtaining a comprehensive set of skills by bypassing the whole process and instead develop a new framework to learn job titles directly through job descriptions (JDs) without the need for the skill extraction pipeline. We introduce job description aggregation network, which reweights each segment of the JD by their importance and then aggregates them into a unified JD representation. Our contributions are as follows:
• Our JD-based method outperforms all previous skill-based approaches in both in-domain and outof-domain settings, achieving up to a 1.8% and 1.0% absolute performance gain.
• Our ablation study shows that the ability to reweight segments according to their importance afforded by our model is critical to its accuracy.
• We show that our approach can implicitly learn the information about the underlying skills associated with job titles.
1319

heading: 2 Our Proposed Method
text: An overview of the proposed framework is illustrated in Figure 1. Initially, job titles and their respective segmented job descriptions are independently fed into a sentence encoder to obtain their representation. Subsequently, the sentence embeddings are fused through the job description aggregator to acquire a unified representation. Finally, bidirectional contrastive loss is utilized as a training objective to maximize a pairwise similarity between the embedded job title and their respective aggregated job description representation while minimizing others.
The subsections describe the sentence encoder, job description aggregator, and contrastive learning process in detail.

heading: 2.1 Sentence Encoder
text: The sentence encoder follows a dual-encoder architecture (Guo et al., 2018; Feng et al., 2022), in which the job title and description are fed into the encoder separately to generate their respective representations. The job title is fed into the model to obtain a job title embedding h. On the other hand, the job description can be lengthy with some irrelevant parts. Thus, the job description is broken down into sentences and encoded sentence by sentence, resulting in a list of segmented sentence embeddings G = [g1, g2, . . . , gn]. Then, the embeddings are aggregated into a final representation for the description. The sentence segmentation process is explained in Appendix A.

heading: 2.2 Job Description Aggregation Network (JDAN)
text: The job description aggregator is responsible for combining multiple sentence embeddings into a unified representation by weighting the importance
of each sentence. This step is important because job descriptions often contain information not directly related to the corresponding job titles such as location and salary. Inspired by (Wu et al., 2021; Zhang et al., 2023), we create an additional learnable token g<CLS> to represent the summarized token and prepend it to the sequence of sentence embedding G = [g<CLS>, g1, g2, . . . , gn]. A Layer Normalization and a shallow transformer encoder are then applied to D to obtain a list of learned representation D = [d<CLS>, d1, d2, . . . , dn].
D = TransformerEncoder(LayerNorm(G)) (1)
The learned summarized token d<CLS> is then fed through three MLP layers with ReLU activation to obtain a final unified representation f .

heading: 2.3 Bidirectional Contrastive Learning
text: The bidirectional contrastive learning minimizes the following training objective function:
Li = −(log e sim(hi,fi)/τ∑N
j=1 e sim(hi,fj)/τ
+ log e sim(hi,fi)/τ∑N
j=1 e sim(hj,fi)/τ
)
(2) where hi is the ith job title embedding, fi is ith the unified job description embedding, τ is the temperature scaling parameter, and N is the batch size. Instead of only maximizing the similarity between the job title embedding hi and its respective job description embedding fi while minimizing the similarities of other pairs sim(hi, fj), the objective function also further introduces the second term to minimize the similarity of the job description embedding fi to other job titles sim(hj , fi). This could be seen as an extension of SimCSE (Gao et al., 2021) where an additional latter term is included to mitigate the overlooked characteristic of
JAD where a bidirectional relationship (job title to job description and vice versa) exists. (Yang et al., 2019; Feng et al., 2022).

heading: 3 Experimental Setup
text: We benchmarked the performance of our proposed framework under two settings: in-domain and outof-domain.
For the in-domain evaluation, we used our own JTG-Jobposting dataset for training and JTG-Synonym for evaluation. JTG-Jobposting is a private Thai-English job posting dataset consisting of 28,844 job postings from, https:// jobtopgun.com, a renowned recruitment website in Thailand. The postings include job titles, job descriptions, and skills. We performed benchmarking on the JTG-Synonym dataset by posing the problem as a cross-lingual synonym retrieval task where the job titles were used as queries, and all synonyms were used as the candidate pool. Each query was performed on the English and Thai candidate pools separately to calculate the R@5, R@10, and mAP@25. The final metric values were obtained by averaging across every querycandidate-pool pairs. This evaluation protocol was intentionally designed to avoid language bias where a query would prefer a candidate from the same language. This issue will be later discussed in the Section 4.5.
For the out-of-domain evaluation, we used Mycareersfuture.sg (Bhola et al., 2020), a dataset of real-world job postings collected by the Singaporean government as the training set. ESCO (le Vrang et al., 2014), a standardized system of the European Union (EU) for categorizing skills, competencies, qualifications and occupations was used as the validation and test data. The task chosen for benchmarking is job normalization where the goal is to predict the standardized version of each job title. The ESCO job normalization dataset consists of 30,926 unique job titles and 2,675 standardized ESCO occupation labels. We followed (Decorte et al., 2021) and used the standard micro-average of recall at 5, 10 and MRR as metrics.
The summary statistics of the datasets are shown in Table 3. See Appendix C for more details.

heading: 3.1 Implementation Details
text: We compared the performance of our proposed framework to other previously proposed skill-based methods, which are JobBERT (Decorte et al.,
2021), Doc2VecSkill (Zbib et al., 2022), VacancySBERT (Bocharova et al., 2023), and our own skill-based approach. In our approach, we used a dual-encoder to independently encode the job title and the concatenated set of skills corresponding to the title using a specified separator token ("[SEP]" for BERT (Devlin et al., 2019) and "
" for XLM-R (Conneau et al., 2019)). Then, the bidirectional contrastive loss was used to encourage the job title and the respective encoded skills pair to come together while pushing the others away. All skill-based models used keyword matching to extract skills from the job description.
For automatic skill extraction, we used SkillSpan1 to extract keywords from the Mycarrersfuture.sg dataset. On the other hand, as the JTG-job posting dataset contains both English and Thai, we made some modifications by creating a new classifier to extract the skills from the job description by posing the problem as a multi-label classification instead. Additional details for the automatic skill extraction is provided in Appendix D.2.
The experiments were conducted using the pretrained-language model BERT as a sentence encoder (Devlin et al., 2019) on the job normalization task and XLM-R (Conneau et al., 2019) on the synonym retrieval task because JTG-Jobposting contains both Thai and English. The sentence representations of every model were obtained through mean pooling of every token (words) in the sentence. We reported the average with standard deviation from five random seeds. The hyperparameters were obtained through grid search in the validation set. The extended implementation detail of our method is shown in the Appendix B, D.

heading: 4 Results and Analysis

heading: 4.1 Main Results
text: Our method achieved consistent performance improvement over all previous skill-based approaches on both datasets under all metrics, achieving up to 1.8% and 1.0% absolute performance gain on the JTG-Synonym and Job normalization task, respectively (Table 1).
1https://huggingface.co/jjzha/jobbert_ skill_extraction

heading: 4.2 Comparison between Skill-based and JD-based method
text: We found that our framework also outperformed or achieved competitive performance when compared to other skill-based approaches, even with human annotations (Table 2). Surprisingly, our method performed better in skill-based recruiter annotation on the job normalization task but not the JTGSynonym task. This is because skill annotations for the MyCareersFuture.sg dataset might be incomplete due to a communication gap between employers and recruiters (Bhola et al., 2020). In addition, some parts of the JTG-Jobposting dataset also contain implicit information that is only present in the recruiter’s annotated skill and not explicitly mentioned in the JD. For example, a job posting for "Data analyst" is annotated with the skills "Excel", "SQL", and "Python," but these skills do not appear in any part of the JD. Additional examples could be seen in Figure 3 of our Appendix.

heading: 4.3 Probe Analysis on Job Title Embeddings
text: We also conducted an additional analysis on the learned embeddings by linear probing (Alain and Bengio, 2017) (training a linear classifier on top of the embeddings) . We trained classifiers to predict skills from the job title embeddings learned from various methods. As shown in Figure 2, the
result of linear probing on a held-out set of the JTGJobposting dataset (details provided in Appendix D.1) found that the model learned through recruiterannotated skills and JD were on par (25.8 vs 24.5 Top-10 accuracy) which doubled the performance of using XLM-R embedding without finetuning (13.1 Top-10 accuracy), implying that the JD-based method could implicitly understand skill information despite not being trained with one. This offers an explanation on why learning the job title representation from the job description can be more beneficial than from skills as the skill information can be learned implicitly while having access to other additional information.

heading: 4.4 Aggregation Design Choices
text: Next, we analyze our design of the job description aggregator and compare it against three other possibilities (Table 5). Instead of using transformers to aggregate multiple sentence embeddings, we can use mean or max pooling. Another possibility is to use the encoder to encode the entire job description (Document Level). Our proposed method outperforms the others, highlighting the importance of having a segmented sentence representation and
weighting mechanism for each sentence. We provide examples of how the first attention layer selects important parts of the job description in Figure 6 and Section E of our Appendix.

heading: 4.5 Cross-lingual Evaluation
text: Studies have pointed out the problem of language bias in textual embeddings (Roy et al., 2020; Yang et al., 2021). Embeddings from the same language are generally closer together compared to their cross-lingual counterparts. As a result, a query in Thai will prefer Thai candidates over English ones, and vice versa. To avoid this bias, the candidate pool was divided into Thai and English pool, and retrieval was done separately. Table 4 shows the retrieval results for different query-candidate-pool pairs. As expected, all models perform better despite the pool being split. Cross-lingual setups are generally more challenging. However, our method consistently outperforms others in every setting.

heading: 5 Conclusion
text: In this paper, we propose a framework for learning the semantic similarity of job titles through job descriptions, bypassing the need for a complete set of skills. The job description aggregator and bidirectional contrastive loss are also introduced to handle the nature of lengthy job descriptions and the two-way relationship between the job title and its description. Our results show that our approach achieves superior performance over the previous state-of-the-art skill-based methods.

heading: 6 Limitations
text: The limitations of our work are as follows: • Our framework is limited to information only available in the job description. Thus, in some cases, our performance might be sub-optimal than recruiter annotations, which could provide information that is not explicitly mentioned in the job description. • Our job description aggregator requires the description to be segmented. This could be challenging when applied to languages other than English. • The job description aggregator is not designed for encoding the entire job description, which does not guarantee that our job description aggregator can be further used in downstream tasks that require job description embedding.

heading: 7 Ethics Statement
text: An inclusion of the job description for learning the job title could induce gender and age bias into job recommendations and search results. This might affect the fairness and inclusiveness of the job matchmaking process. (Saxena and Jain, 2021; Seyedsalehi et al., 2022).

heading: 8 Acknowledgements
text: This work is supported in part by JOBTOPGUN, job postings and recruitment platform in Thailand. We also would like to thank the Chulalongkorn Computational Molecular Biology Group (CMB@CU) for providing additional computational resources.

heading: A Data Pre-processing
text: For the JTG-Jobposting dataset, we used the field "Job_Description" as a job description, and a heuristic algorithm was then applied to segment it by splitting them based on bullet points, hyphens, and numbering using regular expression.
For the MyCareersFuture.sg dataset, we concatenated the fields "Role & Responsibilities" and "Job Requirement" to represent the job description as suggested in their work. However, since stop words and punctuations have been removed from this data, we used a punctuation restoration model2 and then apply NLTK sentence segmentation for segmenting the job description.

heading: B Hyperameters Tuning
text: Table 6 shows the hyperparameters chosen for grid search. The search was based on the validation performance (mAP@25 or MRR). The AdamW optimizer was used with a linear warm-up for 10% of the training steps with a batch size of 16. Every model was trained for ten epochs on the JTGSynonym task and five epochs on the job normalization task. We calculated R@5, R@10, MRR, and mAP@25 using the trec_eval Python package (Van Gysel and de Rijke, 2018). Every experiment was conducted using PyTorch (Paszke et al., 2019) and NVIDIA RTX 3090GPU with 24GB memory.
All models ended up with a temperature parameter, τ , of 0.05. For our skill-based approach, an initial learning rate of 3e-5 and 1e-5 was used in the JTG-Synonym and job normalization task, respectively. For our JD-based approach, an initial learning rate of 1e-5 and 3e-5 were used in the JTGSynonym and job normalization task, respectively. The number of transformer layers used in the job description aggregator was 4.
2https://huggingface.co/felflare/ bert-restore-punctuation

heading: C Detailed Dataset Description
text: C.1 JTG-Jobposting Dataset statistics of the JTG-Jobposting dataset are shown in Table 3. Overall, the dataset statistics are very similar to the Mycarrersfuture.sg dataset, though the former has a much higher number of distinct skills. This is because skill in the JTG-Jobposting dataset is more relaxed compared to Mycarrersfuture.sg where a predefined set of skills is available. Some data examples from JTGJobposting are shown in Figure 4. It consists of the following fields:
• "Position_Name" : a job title of the job posting
• "Skill_Hashtag" : the skill tags assigned by the recruiter.
• "Job_Description": a job description for the job posting.
C.2 JTG-Synonym The JTG-Synonym is a synonym list that includes different variants of the same job title. We split the list into validation and test of size 2,000 and 4,420, respectively. Thai and English titles were kept separated. Examples are shown in Figure 5. The statistics of the JTG-Synonym are shown in Table 7.

heading: D Extended Implementation Detail
text: This subsection further describes our method and competing approaches. The model weight and inference code are available at https://github. com/SLSCU/JD-agg-network.
D.1 Linear Probing We applied linear probing by freezing the whole model except for the last feedforward layer. The performance of linear probing was evaluated on another set of JTG-Jobposting data to ensure no overlapping with the training data. The objective of the task is to predict the appropriate skills given
the job titles. The dataset contains 6,861 training samples and 2,000 testing samples consisting of 157 classes (skills). Due to the sparsity of skill labeling, we evaluated the performance using top10 accuracy. The experiment was conducted using job title embeddings from three sources: our skillbased method with recruiter annotation, JD-based method, and XLM-R without fine-tuning.
D.2 Skill Extraction Model
The model used for skill extraction is mUSE (Yang et al., 2020) followed by a 2-layer MLP with crossentropy loss as the objective. During inference, we select the top 10 scores as candidates for representing the skill tags for each job posting. The model was trained on another separate set of the JTG-Jobposting dataset containing 12,240 samples, totaling 35,107 skills, that do not overlap with the
original JTG-Jobposting dataset.
D.3 Comparison against other methods
Since the performance of the skill-based method of the JTG-Jobposting was not available, we reimplemented the baselines using the following configurations:
• JobBERT: We followed (Decorte et al., 2021) by randomly selected five samples from a distribution defined by the frequency distribution of skills in the training corpus, raised to the power of 3/4 for training using the skip-gram technique. We used a batch size of 64 and a learning rate of 5e-6.
• Doc2VecSkill: We reimplemented this baseline by aggregating the skill set in each job title, and then Doc2vec was used to convert
this set into auxiliary skill embeddings. The Doc2vec model was trained for 100 epochs with a dimension size of 768. Then, we matched the auxiliary skill embeddings with their embedded job titles using the cosine similarity loss. We used a batch of 64 and a learning rate of 3e-5.

heading: E Qualitative Results
text: Figure 6 shows examples of attention maps in the first attention layer from the job description aggregator. It was found that the aggregator could correctly attend to sections with high importance and ignore the sentences unrelated to the job title. For example, in row 2 of Figure 6, the sentences "managing document of the developed software" and "Being studious, self-taught, responsible, and self-improvement" were mostly ignored while the sentence "Design and develop CRM Web Application ..." was strongly attended. The observation also held even when the first line was not the most informative sentence (example number 3 and 4).

heading: F Design choice for our skill-based method
text: We explored different design choices for combining multiple skills to train our skill-based method. These included averaging, maximizing the output of embedded skills, and concatenating the skill list. The results in Table 8 suggest that concatenation, our final choice, performed best.

heading: G Design choice for the sentence representation aggregation
text: We explored different approaches for representing the sentence embedding from a sequence of tokens by comparing token aggregation using an average with directly using the [CLS] token. It was found that averaging the tokens in the sentence yielded marginal performance improvement over the usage of the [CLS] token.

heading: Future Work
text: Overall, the dataset statistics are very similar to the Mycarrersfuture. sg dataset, though the former has a much higher number of distinct skills. This is because skill in the JTG-Jobposting dataset is more relaxed compared to Mycarrersfuture. sg where a predefined set of skills is available. Some data examples from JTGJobposting are shown in Figure 4. It consists of the following fields:
• "Position_Name" : a job title of the job posting
• "Skill_Hashtag" : the skill tags assigned by the recruiter. • "Job_Description": a job description for the job posting. C. 2 JTG-Synonym The JTG-Synonym is a synonym list that includes different variants of the same job title. We split the list into validation and test of size 2,000 and 4,420, respectively. Thai and English titles were kept separated. Examples are shown in Figure 5. The statistics of the JTG-Synonym are shown in Table 7