SlowGuess/NAACL_2022_OCR huggingface dataset:
Jun Zhu, Céline Hudelot
Laboratoire Mathématiques et Informatique pour la Complexité et les Systèmes
CentraleSupélec,UniversitéParis-Saclay
Gif-sur-Yvette, France
{jun.zhu, celine.hudelot}@centralesupelec.fr
Works on learning job title representation are mainly based on Job-Transition Graph, built from the working history of talents. However, since these records are usually messy, this graph is very sparse, which affects the quality of the learned representation and hinders further analysis. To address this specific issue, we propose to enrich the graph with additional nodes that improve the quality of job title representation. Specifically, we construct Job-Transition-Tag Graph, a heterogeneous graph containing two types of nodes, i.e., job titles and tags (i.e., words related to job responsibilities or functionalities). Along this line, we reformulate job title representation learning as the task of learning node embedding on the Job-Transition-Tag Graph. Experiments on two datasets show the interest of our approach.
The learning of job title (referred to as job for short)  ${}^{1}$  representation has received much attention in the recruitment field because the learned representation is beneficial to various tasks, such as job recommendation (Dave et al., 2018; Liu et al., 2019b), job title benchmarking (Zhang et al., 2019), and job mobility prediction (Zhang et al., 2021; Zhu et al., 2021). However, in practice, learning a good representation is challenging for the following reasons: (i) Noisy data: job title data is noisy due to personal subjective reasons (i.e., spelling errors) or objective reasons (i.e., resume parsers are not perfect). (ii) Messy data: job titles are messy because people have different ways of thinking, and naming conventions vary by company and industry. For example, there are many alternative job titles for the same position, e.g., "purchasing clerk" and "buyer". Another problem is that due to the ambiguity of certain terms, they can refer to different positions in different contexts, e.g., "registered nurses
sandwich rehab" and "sandwich maker". For these reasons, standard semantic-based approaches that aggregate (e.g., mean or sum) word representations to get job title semantic representation may lead to mismatches. Moreover, these methods ignore hidden relationships between job titles, e.g., titles in the same resume may be similar. (Dave et al., 2018; Zhang et al., 2019) learn representations from graphs. They create graphs from career trajectories, where nodes represent job titles and edges represent job transitions. Then they design different loss functions to embed the nodes into a low-dimensional space. However, the generated graphs are usually sparse due to the above reasons, limiting the performance of graph-based methods. Standardizing job titles before generating graphs can alleviate the sparsity issue to a certain extent, but at the cost of losing some information. To tackle these challenges, we propose to enrich graphs with structured contextual information and learn job title representations through network embedding methods. More specifically, inspired by domain-specific Named Entity tags (i.e.,  $\text{RESponsibility}$  and  $\text{FUNction}$ ) proposed in (Liu et al., 2019a), we treat the job title as a combination of responsibilities, functionalities, and other additional information. Words related to responsibility and functionality are defined as tags. We assume that job titles with the same tag describe similar job responsibilities or functionalities, making them more likely to have similar representations. Along this line, we construct Job-Transition-Tag Graph, a heterogeneous graph containing two types of nodes, i.e., job titles and tags, which carries more information, thereby alleviating the sparsity problem.
A graph/network is represented as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  with node set  $\mathcal{V}$  and edge set  $\mathcal{E}$ . Nodes and edges
In this paper, we use job title and job interchangeably.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2133 - 2140 July 10-15, 2022 ©2022 Association for Computational Linguistics
can optionally have a type, so a graph can be homogeneous or heterogeneous. In the recruitment field, the career trajectory of talents can be represented by graphs. Formally, consider a job seeker set  $\mathcal{U}$  and their working history set  $\mathcal{H} = \{H^u\}_{u\in \mathcal{U}}$  where the working history of each  $u$  is represented as a sequence of  $n$  work records ordered by time  $H^{u} = \{J_{1},\ldots ,J_{n}\}$ . The  $i$ -th record  $J_{i}$  is denoted by  $(j_i,p_i)$ , indicating that  $u$  is engaged in a position (titled  $j_{i}$ ) during the  $p_i$  period. The set of job titles  $j_{i}$  that occurred in  $\mathcal{H}$  is denoted as  $\mathcal{I}$ . Based on  $\mathcal{H}$ , Job-Transition Graph (Figure 1a) can be constructed, which is formally defined as:
Definition 1 (Job-Transition Graph) is defined as a directed homogeneous graph  $\mathcal{G}^{jj} = (\mathcal{J},\mathcal{E}^{jj})$  generated from  $\mathcal{H}$ , where  $\mathcal{J}$  is a set of job titles, and the edge  $e_{xy}^{jj}\in \mathcal{E}^{jj}$  represents the job transition from the former job  $j_{x}$  to the next job  $j_{y}$ .
2.1.1 Learning from Job-Transition Graph: An Overview
$\mathcal{G}^{jj}$  is often used for job title representation learning tasks. The current procedure is first to build a  $\mathcal{G}^{jj}$ , and then learn job title representation from it. More specifically, (Dave et al., 2018) first builds  $\mathcal{G}^{jj}$  and the other two graphs. Then, the Bayesian personalized ranking and margin-based loss functions are used to learn job title representations from graphs. Job2Vec (Zhang et al., 2019) constructs a  $\mathcal{G}^{jj}$ , where the node denotes a job title affiliated with the specific company, and a multi-view representation learning method is proposed. (Zhang et al., 2021) adds company nodes in  $\mathcal{G}^{jj}$  to build a heterogeneous graph. Then they use a graph neural network to represent the nodes. As mentioned in Section 1, the job title and job transition data are messy. Therefore,  $\mathcal{G}^{jj}$  may be sparse (Zhang et al., 2019), which we will further prove in Section 3.1. In order to alleviate this issue, a simple method is to standardize job titles and then construct a normalized and denser graph based on the standardized job titles. For example, (Dave et al., 2018) normalizes titles by using Carotene (Javed et al., 2015), (Zhang et al., 2019) aggregates titles by filtering out low frequency words, and (Zhang et al., 2021) unifies titles according to IPOD (Liu et al., 2019a). However, the standardization of job titles may lose some specific information. Furthermore, these methods either ignore the semantic information contained in job titles (Dave et al., 2018; Zhang et al., 2021) or separate the semantic information from the graph topology (Zhang et al., 2019).
2.1.2 Job Title Composition
Generally, a job title consists of three parts (Liu et al., 2019a; Zhang et al., 2019): (i) Responsibility: describes the role and responsibility of a position from different levels (e.g., director, assistant, and engineer). (ii) Functionality: describes the business function of a position from various dimensions (e.g., sales, national and security). (iii) Additional Information: contains personal-specific information. We denote the words related to responsibility and functionality as tags, and they form a tag set  $\mathcal{T}$ . These tags are the essence of the job title and provide important information about the position. However, few works directly include this information in the representation learning scheme. In this paper, we consider these tags when generating graphs. These tags can alleviate the graph sparsity problem of  $\mathcal{G}^{jj}$  and provide additional information for the learning of job title representations.
2.2 Proposed Graphs
In order to address the sparsity issue of  $\mathcal{G}^{jj}$ , we consider adding more information when generating graphs, i.e., tags related to job responsibilities or functionalities, driven by the job title composition. Along this line, we define various types of graphs:
Definition 2 (Enhanced Job-Transition Graph) is based on  $\mathcal{G}^{jj}$  with additional enhanced edges. It is defined as  $\mathcal{G}_E^{jj} = (\mathcal{J},\mathcal{E}^{jj}\cup \mathcal{E}_E^{jj})$  , where  $\mathcal{E}_E^{jj}$  is a set of enhanced edges. More specifically, if  $j_{x}$  and  $j_{y}$  share a tag  $w$  , then we add a bi-directional edge between them, i.e.,  $e_{xy}^{jj}$  and  $e_{yx}^{jj}$
As shown in Figure 1b, red dashed lines represent additional enhanced edges, e.g., "purchasing manager" shares the tag "purchasing" with "purchasing clerk", so we add edges between them.
Definition 3 (Job-Tag Graph) is defined as a heterogeneous graph  $\mathcal{G}^{jt} = (\mathcal{I}\cup \mathcal{T},\mathcal{E}^{jt})$ , with job titles and tags, two node types.  $\mathcal{E}^{jt}$  is a set of bi-directional edges between a job title and a tag, representing the "has/in" relationship.
An example of the "has/in" relationship is given in Figure 1c (i.e., the green line), where the job title "automotive technician" has a tag "automotive", and "automotive" is in "automotive technician". In order to aggregate more information, we further combine Job-Transition Graph and Job-Tag Graph to build Job-Transition-Tag Graph:
Definition 4 (Job-Transition-Tag Graph) is defined as a heterogeneous graph  $\mathcal{G}^{jtj} = (\mathcal{J}\cup$
(a) Job-Transition graph.
(c) Job-Tag graph.
Figure 1: Examples of four types of graphs with blue/green circles representing job titles-tags. Black lines represent job transitions, red dotted lines represent additional enhanced edges, and green lines represent "has/in" relationships.
(b) Enhanced Job-Transition graph.
(d) Job-Transition-Tag graph.
$\mathcal{T}, \mathcal{E}^{jj} \cup \mathcal{E}^{jt}$  with job titles and tags, two node types, and transition and "has/in", two edge types.
Inspired by the achievements of network embedding models in the node representation learning problem (Hamilton et al., 2017), we apply different network embedding models to learn job title representation from the graphs defined above.
We evaluate the proposed job title representation learning scheme through (i) a node classification task (i.e., job title classification) and (ii) a link prediction task (i.e., next-job prediction). This section will first describe the two datasets used and experimental settings, followed by discussing the results.
CareerBuilder12 (CB12): an open dataset from a Kaggle competition. It contains a collection of working experiences represented by sequences of job titles. For the node classification task, we use AutoCoder to assign a SOC 2018 to each job title. The labeling details are given in Appendix A.1. Randstad: a private French resume dataset provided by Randstad company, where each resume is parsed into multiple sections, e.g., PersonalInformation, EducationHistory and EmploymentHis
tory. An example is given in Figure 3 of Appendix. Graphs are built from EmploymentHistory section.
Both datasets use a tree-like taxonomy, as described in Appendix A.1 and A.2. For example, from root class to leaf class, the  $CB12$  taxonomy is organized as MajorGroup  $\rightarrow$  MinorGroup  $\rightarrow$  BroadGroup  $\rightarrow$  DetailedOccupation. Consistent with the reality of the recruitment market, some occupations rarely appear in both datasets. To balance the data, we filter out occupations with fewer than 200 occurrences, i.e., MinorGroup for  $CB12$  and JobGroup for Randstad. Also, for graph construction, we remove working histories with less than two work records, i.e.,  $|H^u| < 2$ .
#C	#W	|J|	|Ejj|	|EjE|	|Ejt|
CB12	16	1,682	9,216	20,640	6,477,819	22,149
Randstad	18	2,303	12,864	36,722	6,663,267	22,897
Table 1: Statistics of datasets and corresponding graphs, #C represents the number of categories, and #W represents the vocabulary size for node one-hot encoding.
In order to generate tags, we first tokenize titles into tokens and remove stopwords, numbers, and punctuation. Then, we use the Top-200 tokens that appear most frequently in job titles and belong to IPOD (Liu et al., 2019a) as tags. The details of tag generation are given in Appendix A.3. We assign the one-hot encoding of the corresponding title for each title node as the node feature. The vocabulary set is obtained by filtering words with a frequency of 1 from the tokenized job titles. Statistics for
$^{2}$ https://www.kaggle.com/c/  
job-recommendation  
 $^{3}$ http://www.onetsocautocoder.com/plus/onetmatch
N2V	GCN	GAT	M2V	RGCN	HAN	W2V	BERT
CB12	Gjjj	0.206/0.360	0.576/0.688	0.568/0.664	0.154/0.334	0.524/0.637	0.670/0.747	0.713/0.767	0.688/0.719
GjE	0.599/0.714	0.628/0.720	0.692/0.759	0.571/0.688	0.591/0.701	0.698/0.781
Gjt	-	-	-	0.588/0.692	0.687/0.752	0.703/0.766
Gjjt	-	-	-	0.588/0.692	0.703/0.766	0.742/0.797
Randstad	Gjjj	0.201/0.304	0.520/0.616	0.529/0.593	0.166/0.282	0.388/0.536	0.592/0.665	0.595/0.671	0.580/0.609
GjE	0.523/0.623	0.484/0.621	0.607/0.677	0.469/0.585	0.452/0.580	0.607/0.689
Gjt	-	-	-	0.590/0.665	0.552/0.643	0.572/0.663
Gjjt	-	-	-	0.590/0.665	0.600/0.678	0.641/0.708
Table 2: Job title classification results (Macro-F1/Micro-F1). The score in bold is the best among all methods applied to all graphs, and the scores underlined are the best in all graphs of each method. For  $M2V$ , we report the best results obtained by the meta-path Job-Tag-Job.
datasets and graphs are summarized in Table 1. We can observe that  $\mathcal{G}^{jj}$  (i.e.,  $|\mathcal{I}|$  and  $|\mathcal{E}^{jj}|$ ) are sparse.
For job title classification task: we classify job titles into root categories in this work, i.e., Major-Group for CB12 and JobClass for Randstad. We randomly split the data into training/validation/test sets with a ratio of  $60\% / 20\% / 20\%$ .
For next-job prediction task: we treat it as a link prediction task on Job-Transition Graph to predict whether there exists an edge (transition) between two nodes (job titles). We keep the same split ratio on positive/negative edges, where negative edges are randomly picked from unconnected node pairs (i.e., the same size as positive edges).
We evaluate the performance of our proposed learning scheme against the following baselines. A detailed description of these baselines is provided in Appendix A.4.
- Homogeneous: Node2Vec (N2V) (Grover and Leskovec, 2016), GCN (Kipf and Welling, 2017) and GAT (Velickovic et al., 2018).
- Heterogeneous: Metapath2Vec (M2V) (Dong et al., 2017), RGCN (Schlichtkrull et al., 2018) and HAN (Wang et al., 2019b).
- Semantic-based: Word2Vec (W2V) (Le and Mikolov, 2014) and BERT (Devlin et al., 2019).
Our implementation is based on the DGL package (Wang et al., 2019a). In both tasks, for unsupervised methods, node representations are learned from the entire dataset. Then train the logistic regression classifier on both the training and validation sets. Each semi-supervised model is trained on the training set, and the parameters are optimized on the validation set. The final performance is evaluated on the test set. To ensure fairness, we
keep the same data split for both methods, repeat each prediction experiment ten times, and report the average performance scores (i.e., Macro-F1 and Micro-F1 for job title classification and AUC for next-job prediction). For details of other parameter settings, see Appendix A.5.
3.3.1 Job Title Classification
Table 2 summarizes the best results of all methods on different graphs. We have the following observations: (i) Among all graphs, all models generally have the lowest scores on  $\mathcal{G}^{jj}$  because this graph is often sparse and can only provide limited information. (ii) All models perform better on  $\mathcal{G}_E^{jj}$  (except Macro-F1 of GCN) than  $\mathcal{G}^{jj}$ , which shows that the enhanced edges provide additional information. One interpretation for enhanced edges is adding semantic information, i.e., two job titles are more likely to be similar if they share the same word, represented by edges from the graph perspective. (iii) The heterogeneous models perform well on our proposed  $\mathcal{G}^{jtj}$ , which indicates that the added tag nodes can effectively improve the quality of representation. Note that we did not apply homogeneous methods to  $\mathcal{G}^{jtj}$ , but the results on  $\mathcal{G}_E^{jj}$  prove that the information given by tags is useful. (iv) The models with attention mechanisms outperform the models without attention, demonstrating that the attention mechanism is good at capturing important information from noisy graphs.
3.3.2 Next-Job Prediction
We further evaluate the learning scheme using next-job prediction, which can be viewed as a link prediction task to predict whether a position will be recommended as the next-job. For unsupervised methods, edge features are represented by applying binary operators (Grover and Leskovec, 2016)
$^{4}$ Source code will be available at https://github.com/zhujun81/Job_title Representation.
N2V	GAN	M2V	HAN (Dot)	W2V (Dot)	BERT (Dot)	W2V (Hadamard)	BERT (Hadamard)
Gjj	0.564	0.704	0.548	0.685
GjjE	0.692	0.789	0.593	0.792	0.763	0.477	0.777	0.840
Gjt	-	-	0.604	0.768
Gj t j	-	-	0.604	0.833
Table 3: Next-job prediction results (AUC) on CB12. The bold score is the best among all methods, and the underlined score is the second-best.
on node pairs, and then the best binary operator is selected based on the validation set, while the dot product is used for semi-supervised methods. The results on CB12 given in Table 3 show the promising results of our proposed graphs. Like job title classification, the scores of all network embedding methods, i.e.,  $N2V$ ,  $GAN$ ,  $M2V$  and  $HAN$  better on  $\mathcal{G}_E^{jj}$  compared to  $\mathcal{G}^{jj}$ , and the heterogeneous models perform best on  $\mathcal{G}^{jtj}$ . Such results further demonstrate the effectiveness of our proposed method for constructing graphs, whether adding additional information based on tags (i.e.,  $\mathcal{G}_E^{jj}$ ) or directly adding tags to the graph (i.e.,  $\mathcal{G}^{jt}$  and  $\mathcal{G}^{jtj}$ ). BERT using Hadamard operator performs best, followed by  $HAN$  on  $\mathcal{G}^{jtj}$  with a slight difference of 0.007. However, when we use the dot product used in  $HAN$  to obtain edge features for BERT, the AUC of BERT drops sharply to 0.477, while  $W2V$  only drops a little to 0.763. We will discuss such results in future work. Overall, the results of link prediction also demonstrate the effectiveness of our proposed graphs.
3.3.3 Visualization
For a more intuitive comparison, we select five occupations and then visualize the job title representations learned by HAN in Figure 2, with each color corresponding to an occupation category. Overall, the representations learned by HAN on all graphs are clustered into groups. However, when considering tags, representations are easier to be subdivided further in each category. For example, in Figure 2d, the orange occupation can be further divided into three sub-clusters, which proves that adding tag nodes can help capture more detailed information and make the learned representation more informative. This detailed information helps further categorize occupations, as we only classify job titles into the root category (i.e., MajorGroup) in this work.
This paper first proposes to enrich Job-Transition Graph commonly used in job title representation
(a) HAN  $(\mathcal{G}^{jj})$
(b) HAN  $(\mathcal{G}_E^{jj})$
(c) HAN  $(\mathcal{G}^{jt})$
(d) HAN  $(\mathcal{G}^{jtj})$
Figure 2: Visualization of representations (CB12). Healthcare support (green), Healthcare practitioners and technical (blue), Architecture and engineering (purple), Office and administrative support (orange) and Transportation and material handling (red).
learning tasks by adding tag-related information or directly adding tag nodes, and then learn job title representations through network embedding methods. This enhancing method can alleviate the sparsity problem in Job-Transition Graph, thereby improving the quality of learned representations, as demonstrated in the experimental results of job title classification and job title classification. In future work, we would like to explore why the Hadamard operator and dot product lead to such different link prediction results for BERT. Furthermore, other research lines are (i) considering edge weights when learning from graphs, (ii) classifying job titles into different occupational levels, and (iii) improving the tag generation approach.
This work is supported by the Randstad research chair in collaboration with MICS Lab, Centrale-Supélec, Université Paris-Saclay. We would like to thank the Mésocentre  $^{5}$  computing center of CentraleSupélec and École Normale Supérieure Paris-Saclay for providing computing resources.
<sup>5</sup>http://mesocentre.centralesupelec.fr/
Vachik S. Dave, Baichuan Zhang, Mohammad Al Hasan, Khalifeh AlJadda, and Mohammed Korayem. 2018. A combined representation learning approach for better job and skill recommendation. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 1997-2005. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pages 135-144. ACM.
Jean-Philippe Fauconnier. 2015. French word embeddings.
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 855-864. ACM.
William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Representation learning on graphs: Methods and applications. ArXiv preprint, abs/1709.05584.
Faizan Javed, Qinlong Luo, Matt McNair, Ferosh Jacob, Meng Zhao, and Tae Seung Kang. 2015. Carotene: A job title classification system for the online recruitment domain. In 2015 IEEE First International Conference on Big Data Computing Service and Applications, pages 286-293. IEEE.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Quoc V. Le and Tomás Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June