SlowGuess/ACL_2020_OCR huggingface dataset:
Liting Liu $^{1}$ , Jie Liu $^{2*}$ , Wenzheng Zhang $^{2}$ , Ziming Chi $^{2}$ , Wenxuan Shi $^{1}$ , Yalou Huang $^{1}$
<sup>1</sup>College of Software, Nankai University, Tianjin, China
$^{2}$ College of Artificial Intelligence, Nankai University, Tianjin, China
{liu_liting, wzhang}@mail.nankai.edu.cn
zimingchi@163.com
{jliu, shiwx, huangyl}@nankai.edu.cn
Writing a good job posting is a critical step in the recruiting process, but the task is often more difficult than many people think. It is challenging to specify the level of education, experience, relevant skills per the company information and job description. To this end, we propose a novel task of Job Posting Generation (JPG) that is cast as a conditional text generation problem to generate job requirements according to the job descriptions. To deal with this task, we devise a data-driven global Skill-Aware Multi-Attention generation model, named SAMA. Specifically, to model the complex mapping relationships between input and output, we design a hierarchical decoder that we first label the job description with multiple skills, then we generate a complete text guided by the skill labels. At the same time, to exploit the prior knowledge about the skills, we further construct a skill knowledge graph to capture the global prior knowledge of skills and refine the generated results. The proposed approach is evaluated on real-world job posting data. Experimental results clearly demonstrate the effectiveness of the proposed method<sup>1</sup>.
Writing high-quality job postings is the crucial first step to attract and filter the right talents in the recruiting process of human resource management. Given job descriptions and basic company information, the key to the job posting is to write job requirements, which requires to specify professional skills properly. Both too many or few requirements may lead to negative impacts on talent recruiting. Because of the extremely large number of job positions and varieties of professional skills, a lot of
(Basic Information
Position: Market Researcher company scale:  $1000\sim 10000$
1. Assist the General Manager in sourcing travel industry news and in conducting product research and analysis.
2. Facilitate effective communication between the market research and user experience teams.
3. Translate key industry texts and compose newsletters for internal communication.
Job Requirement
1.  $3+$  years of research experience at investment banks.
2. Strong research, data analysis and communication skills.
3. Proficient user of Microsoft Suite/G Suite.
Figure 1: An example of automatic job posting.
companies have to pay much cost in this step to win in the war of talents.
To this end, we propose the task of Job Posting Generation (JPG) in this paper, and we cast it as a novel conditional text generation task that generates the job requirement paragraph. Exploiting the ubiquitous job posting data, we aim to automatically specify the level of necessary skills and generate fluent job requirements in a data-driven manner, as shown in Figure 1.
Although the JPG task is of great significance, the complexity of it poses several key challenges: 1) Generating job requirements needs to not only produce overall fluent text but also precisely organize the key content like skills and other information, which is very difficult to current neural systems. Especially, the long-text to long-text generation easily leads to information missing (Shen et al., 2019). 2) The key points of job descriptions and the skills of job requirements are complex many-to-many relations, which makes the mapping learning very difficult. 3) How to exploit the global information among the heterogeneous relations between basic company information and the professional
$^{1}$ https://github.com/NKU-IPLab/SAMA
3096
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3096-3104
skills across the whole dataset is of great importance to generate high-quality job requirements.
To address these challenges, we focus on the richness and accuracy of skills in generated job requirements and propose a global Skill-Aware Multi-Attention (SAMA) model for JPG task. Specifically, we devise a two-pass decoder to generate informative, accurate, and fluent job requirement paragraph. The first-pass decoder is to predict multiple skills according to the job description, which is a multi-label classification task (Zhang and Zhou, 2014). The second-pass decoder is to generate a complete text according to the predicted skill labels and the input text. Moreover, we build a skill knowledge graph to capture the global information in the whole job posting dataset in addition to the local information provided by the input. Through the skill knowledge graph, our model obtains the global prior knowledge to alleviate the misusing of skills. Extensive experiments are conducted to evaluate our model on real-world job posting data. The result demonstrates the effectiveness of the proposed method.
- We propose a novel task of job posting generation that is defined as the conditional generation given a job description and basic company information to generate a job requirement.
- A data-driven generation approach SAMA is proposed to model the complex mapping relationships and generate informative and accurate job requirements.
- We build a real-world job posting dataset and conducte extensive experiments to validate the effectiveness and superiority of our proposed approach.
2 Data Description
We collect a job posting dataset from a famous Chinese online recruiting market, across a period of 19 months, ranging from 2019 to 2020. There are 107,616 job postings in total. After removing repetitive and too short job postings, 11,221 records are selected. This dataset is collected from 6 different industry domains. The detailed statistics of the dataset are illustrated in Table 1.
Considering the importance of the skills for JPG, we select 2000 records and manually tag the skills in these records. Then we train a word-level LSTM-
training	validation	testing
Internet	2055	509	687
Consumer goods	1153	292	356
Real Estate	969	220	276
Finance	1477	366	463
Automobile	997	282	296
Medical	397	94	115
Table 1: The statistics of the dataset.
CRF model (Huang et al., 2015) to recognize the skills in the whole dataset.
We also keep the basic information, i.e., job position and company scale information, for the reason that they are the critical attributes of job postings that have impacts on the level of skills.
In order to capture the global prior knowledge of skills, we construct a skill knowledge graph according to the semantic relations of entities in the job postings. As shown in Figure 2, there are three types of entities, i.e., skill, company scale, and job position. The entities of skills are divided into two types, generic skills (denoted by G) and professional skills (denoted by P), according to the number of occurrences. The relation N.T.M. (need-to-master) exists between job position entity and skill entity. Besides, the relation IN exists between company scale entity and skill entity. For example, jobseeker who is seeking for a programmer position in a company of 10 to 100 people needs to master the professional skill C++, then there exist three triplets, (programmer, N.T.M., C++, ([10, 100], IN, C++) and (C++, type, P).
Figure 2: An example of the skill knowledge graph.
Let  $\mathcal{D} = \{(B_i,X_i,Y_i)\}_{i = 1}^N$  denote the dataset, where  $X_{i} = (x_{i,1},x_{i,2},\dots,x_{i,m})$  is the word sequence of job description paragraph.  $Y_{i} = (y_{i,1},y_{i,2},\dots,y_{i,n})$  is the word sequence of job requirement paragraph,  $B_{i} = (b_{i}^{p},b_{i}^{s})$  is the basic information,  $b^{p}$  and  $b^{s}$  are job position and company scale information,  $N$  is the size of dataset,  $m$  and  $n$  are the lengths of sequence  $X_{i}$  and  $Y_{i}$ , respectively. The target of the JPG task is to estimate  $P(Y_{i}|X_{i},B_{i})$ , the conditional probability of a
Figure 3: An illustration of the architecture of SAMA that consists of three parts, i.e., skill prediction part, skill refinement part, and job requirement generation part. The skills  $S_{i}$  are predicted given the job description. To consider the global prior knowledge of skills, the skill knowledge graph gives another set of skills  $O_{i}$ , which plays the role of refinement. Finally, SAMA fuses multiple attentions to generate the final job requirement paragraph  $Y_{i}$ .
job requirement  $Y_{i}$  given a job description  $X_{i}$  and basic information  $B_{i}$ .
To tackle the JPG task, we propose a global Skill-Aware Multi-Attention model, named SAMA. Figure 3 shows the overall architecture of SAMA.
Firstly, considering the importance of skill prediction in JPG, we decompose the probability  $P(Y_{i}|X_{i},B_{i})$  into a two-stage generation process, including skill prediction and job requirement paragraph generation:
$$
P \left(Y _ {i} \mid X _ {i}, B _ {i}\right) = P \left(Y _ {i} \mid X _ {i}, S _ {i}, B _ {i}\right) P \left(S _ {i} \mid X _ {i}, B _ {i}\right), \tag {1}
$$
where  $S_{i} = (s_{i,1}, s_{i,2}, \dots, s_{i,l})$  is a skill $^{2}$  word sequence of its corresponding job requirement,  $l$  is the length of  $S_{i}$ . Since  $S_{i}$  and  $B_{i}$  are conditionally independent given  $X_{i}$ , we can derive that  $P(S_{i}|X_{i}, B_{i}) = P(S_{i}|X_{i})$ .
Secondly, for refining the skills, we leverage the global prior information by the skill knowledge graph  $G^{s} = (E_{1},R,E_{2})$  where  $E_{1}$  and  $E_{2}$  are the sets of head and tail entities and  $R$  is the set of relations. Given the basic information  $B_{i}$  and the skill knowledge graph  $G^{s}$ , we obtain a set of skills  $O_{i} = (o_{i,1},o_{i,2},\dots,o_{i,k})$ .
$$
O _ {i} = f \left(B _ {i}, G ^ {s}\right), \tag {2}
$$
where  $f$  is an invertible query function, which can ensure the one to one mapping relation between  $B_{i}$  and  $O_{i}$ .
Thirdly, to fuse the local and global information, the probability  $P(Y_{i}|X_{i},S_{i},B_{i})$  during the text generation process is calculated as:
$$
\begin{array}{l} P (Y _ {i} | X _ {i}, S _ {i}, B _ {i}) = (1 - \lambda) P _ {l o c a l} (Y _ {i} | X _ {i}, S _ {i}, B _ {i}) \\ + \lambda P _ {\text {g l o b a l}} \left(Y _ {i} \mid X _ {i}, S _ {i}, B _ {i}\right), \tag {3} \\ \end{array}
$$
where  $\lambda$  is a hyperparameter that adjusts the balance of two probabilities.
3.1 Job Description Encoder
The input job description word sequence  $X_{i}$  is first transformed into a sequence of word embeddings. To obtain the long-term dependency vector representation, we use a bi-directional LSTM (Schuster and Paliwal, 1997) as the text encoder. The input sequence is transformed into a hidden state sequence  $H = (h_{1}, h_{2}, \dots, h_{m})$  by concatenating the representations of the forward and backward hidden states  $h_t = [\vec{h}_t, \vec{h}_{m-t+1}]$ . Specifically, the initiated encoder hidden state  $h_0$  is a zero vector, and the last encoder hidden state  $h_m$  is used for initiating the skill decoder.
3.2 Skill Prediction
Intuitively, the process of skill prediction is a Multi-Label Classification (MLC) task, which aims to assign multiple skills to each job description. To capture the correlations between skills, inspired by Yang et al. (2018), we view this MLC task as a sequence generation problem.
2The details of how skills are extracted are described in Section 2.
Formally, the skill decoder layer first takes the hidden state  $h_m$  of the encoder as input, then derive a context vector  $C^{st}$  by an attention mechanism (Luong et al., 2015) to help predict the skill labels.
$$
\alpha_ {i} ^ {j} = \frac {\exp \left(g _ {j - 1} ^ {\prime T} W ^ {1} h _ {i}\right)}{\sum_ {i ^ {\prime}} \exp \left(g _ {j - 1} ^ {\prime T} W ^ {1} h _ {i ^ {\prime}}\right)}; C _ {j} ^ {s t} = \sum_ {i = 1} ^ {m} \alpha_ {i} ^ {j} h _ {i}, \tag {4}
$$
where  $W^{1} \in \mathbb{R}^{d \times d}$  is trainable weight matrix,  $d$  is the hidden vector size. Inspired by Yuan et al. (2018), the job description is labelled with multiple skills by generating skills sequence which joins the skills by delimiter  $<\mathrm{SEP}>$  and has an unfixed number of skills (e.g., English  $<\mathrm{SEP}>$  computer science  $<\mathrm{SEP}> \mathrm{c}++$ ). The skill decoder is based on LSTM, whose hidden vector is computed by:
$$
g _ {t} ^ {\prime} = L S T M \left(g _ {t - 1} ^ {\prime}, C _ {t} ^ {s t}\right). \tag {5}
$$
Specifically, the last skill decoder hidden state  $g_{l}^{\prime}$  is used for initiating the text decoder. The skill sequence is finally obtained by a softmax classification over the vocabulary of skills,  $V_{skill}$ . In detail, a non-linear transformation is applied to form the skill decoder semantic representation  $I^{st}$ , and then compute the probability  $P(S_{i}|X_{i},B_{i})$  via:
$$
\begin{array}{c} I _ {j} ^ {s t} = \tanh  (W ^ {2} [ g _ {j} ^ {\prime}; C _ {j} ^ {s t} ]) \\ P (s _ {i, j} | X _ {i}) = \operatorname {s o f t m a x} _ {i} (W ^ {3} I _ {j} ^ {s t} + b ^ {3}), \end{array} \tag {6}
$$
where  $[\cdot ]$  is vector concatenation,  $W^{2}\in \mathbb{R}^{d\times 2d}$ $W^3\in \mathbb{R}^{|V_{skill}| \times d}$  and  $b^{3}\in \mathbb{R}^{|V_{skill}|}$  are parameters.
3.3 Skill Refinement
The process of skill prediction only considers the local information, which results in some misusing of skills. To refine the skill of the generated job requirement, the global information is taken into account by the skill knowledge graph.
The skill entities are divided into G and P as described in Section 2. Here, the basic assumption is that a generic skill appears more frequently than a professional skill among all the job postings, because the professional skill contains more domain characters. We use a hyperparameter  $\theta$  as a threshold to divide the skills entities.
Given the basic information  $B_{i} = (b_{i}^{p}, b_{i}^{s})$ , the set of skills  $O_{i}$  is obtained from the skill knowledge graph by the query function  $f$ . In detail, firstly, we obtain the set of entities that have the "N.T.M." relation with  $b_{i}^{p}$  and the set of entities who have the "IN" relation with  $b_{i}^{s}$ . Secondly, we get the
intersection of the sets obtained in the first step. Finally, we keep the entities whose types are P.
we embed  $O_{i}$  as  $S_{i}^{\prime} = (s_{i,1}^{\prime}, s_{i,2}^{\prime}, \ldots, s_{i,k}^{\prime})$ , and linearly combine it as a skill graph context vector  $C_{j}^{nd}$  by an attention mechanism:
$$
\tau_ {i} ^ {j} = \frac {\exp \left(g _ {j - 1} ^ {T} W ^ {4} s _ {i} ^ {\prime}\right)}{\sum_ {i ^ {\prime}} \exp \left(g _ {j - 1} ^ {T} W ^ {4} s _ {i ^ {\prime}} ^ {\prime}\right)}; C _ {j} ^ {n d} = \sum_ {i = 1} ^ {k} \tau_ {i} ^ {j} s _ {i} ^ {\prime}, \tag {7}
$$
where  $W^4\in \mathbb{R}^{d\times d'}$  are parameters,  $d'$  is the dimensions of the word embeddings. Then a nonlinear transformation is applied to form the graph skill semantic representation  $I^{nd}$ . The probability  $P_{global}(Y_i|X_i,S_i,B_i)$  from  $V_{skill}$  is computed via:
$$
I _ {j} ^ {n d} = \tanh  \left(W ^ {5} \left[ g _ {j}; C _ {j} ^ {n d}; C _ {j} ^ {r d} \right]\right), \tag {8}
$$
$$
P _ {g l o b a l} \left(y _ {i, j} = w \mid X _ {i}, S _ {i}, B _ {i}\right) =
$$
$$
\left\{ \begin{array}{l l} \operatorname {s o f t m a x} _ {i} \left(W ^ {6} I _ {j} ^ {n d} + b ^ {6}\right), & w \in O _ {i} \\ 0, & w \notin O _ {i} \end{array} , \right. \tag {9}
$$
where  $g$  and  $C^{rd}$  will be introduced in next section,  $W^5 \in \mathbb{R}^{d \times (2d + d')}$ ,  $W^6 \in \mathbb{R}^{|V_{skill}| \times d}$ ,  $b^6 \in \mathbb{R}^{|V_{skill}|}$  are trainable parameters.
3.4 Job Requirement Generation
Job requirement generation fuses multiple attention mechanisms from three aspects, job descriptions, predicted skills and skills from skill knowledge graph. The text decoder, based on another LSTM, aims to generate final word sequence. The hidden vector of text decoder is computed by  $g_{t} = LSTM(e_{t - 1},g_{t - 1})$ , where  $e_{t - 1}$  is the word embedding of the final generated target word at time step  $t - 1$ . After obtaining  $g$ , a nonlinear transformation is applied to form the text decoder semantic representation  $I^{rd}$ . The probability  $P_{local}(Y_i|X_i,S_i,B_i)$  is computed via:
$$
I _ {j} ^ {r d} = \tanh  \left(W ^ {7} \left[ e _ {j - 1}; g _ {j}; C _ {j} ^ {r d}; C _ {j} ^ {t h} \right]\right), \tag {10}
$$
$$
P _ {\text {l o c a l}} \left(y _ {i, j} \mid X _ {i}, S _ {i}, B _ {i}\right) = \operatorname {s o f t m a x} _ {i} \left(W ^ {8} I _ {j} ^ {r d} + b ^ {8}\right), \tag {11}
$$
where  $W^7\in \mathbb{R}^{d\times 2(d + d')}$ $W^8\in \mathbb{R}^{|V_{text}| \times d}$ ,  $b^8\in \mathbb{R}^{|V_{text}|}$  are parameters,  $V_{text}$  is the vocabulary of job requirement and  $V_{skill}$  is a subset of  $V_{text}$ , both  $C^{rd}$  and  $C^{th}$  are the context vectors generated by attention mechanisms. Specifically,  $C^{rd}$  is a context vector computed similar as  $C^{st}$  because they
directly take input sequence into account.
$$
\beta_ {i} ^ {j} = \frac {\exp \left(g _ {j - 1} ^ {T} W ^ {9} h _ {i}\right)}{\sum_ {i ^ {\prime}} \exp \left(g _ {j - 1} ^ {T} W ^ {9} h _ {i ^ {\prime}}\right)}; C _ {j} ^ {r d} = \sum_ {i = 1} ^ {m} \beta_ {i} ^ {j} h _ {i}, \tag {12}
$$
where  $W^9\in \mathbb{R}^{d\times d}$
In addition, the skills  $S$  generated by skill decoder are fed into the text decoder to guide the generation process. To obtain  $C^{th}$ , another attention model is leveraged:
$$
\gamma_ {i} ^ {j} = \frac {\exp \left(g _ {j - 1} ^ {T} W ^ {1 0} s _ {i}\right)}{\sum_ {i ^ {\prime}} \exp \left(g _ {j - 1} ^ {T} W ^ {1 0} s _ {i ^ {\prime}}\right)}; C _ {j} ^ {t h} = \sum_ {i = 1} ^ {l} \gamma_ {i} ^ {j} s _ {i}, \tag {13}
$$
where  $W^{10}\in \mathbb{R}^{d\times d^{\prime}}$  are parameters.
The generation probability  $P(Y_{i}|X_{i},S_{i},B_{i})$  is the weighted sum of  $P_{local}(Y_i|X_i,S_i,B_i)$  and  $P_{global}(Y_i|X_i,S_i,B_i)$  as in equation 3. As shown in equation 8 and equation 10, the vector  $C^{th}$  appears explicitly only in  $P_{local}$ , which implies that  $P_{local}$  puts emphasis on the skill prediction, i.e., the local information, while the vector  $C^{nd}$  appears explicitly only in  $P_{global}$ , which indicates that  $P_{global}$  focuses on the skills given by skill knowledge graph, i.e., the global prior knowledge.
In this way, SAMA considers not only the local information from the job description but also the global information from the skill knowledge graph.
The loss function of the model has two parts, the negative log-likelihood of the silver $^3$  skill labels,  $L^S$ , and the gold $^4$  job requirement text,  $L^Y$ :
$$
L ^ {S} = - \sum_ {i = 1} ^ {l} \log P (S | X, B),
$$
$$
L ^ {Y} = - \sum_ {i = 1} ^ {n} \log P (Y | X, S, B), \tag {14}
$$
$$
L = L ^ {S} + \mu L ^ {Y},
$$
where  $\mu$  is a hyperparameter, we give more weight to the loss of gold job requirement. During inference, the outputs of the skill decoder and the text decoder are predicted as follows:
$$
\widehat {S} = \underset {S} {\arg \max } P (S | X, B), \tag {15}
$$
$$
\widehat {Y} = \underset {Y} {\arg \max } P (Y | X, \widehat {S}, B). \tag {16}
$$
For each stage, we obtain the best results by utilizing the greedy search at each step.
In this section, we conduct experiments to verify the effectiveness of SAMA.
Job descriptions and job requirements are tokenized by Pyltp $^5$  word segmenter. Table 1 shows the split of the dataset. There are 468 position entities, 9 scale entities, 31,090 skill entities, and 310,413 relation edges in the skill knowledge graph. The vocabulary of job descriptions contains 14,189 words, the vocabulary of skills contains 3,523 words, and vocabulary the job requirements contains 18,612 words.
4.1.2 Comparison Models
To achieve the comprehensive and comparative analysis of SAMA, we compared it with two kinds of representative models: the standard generation model and the hierarchical generation model.
- S2SA: Seq2Seq with attention (Luong et al., 2015) is a standard generation model.
- DelNet: Deliberation networks model (Xia et al., 2017) is a hierarchical generation model which has a two-pass decoder to generate and polish the same target sequence.
- VPN: Vocabulary pyramid networks (Liu et al., 2019) is a hierarchical generation model which has the multi-pass encoder and decoder to generate a multi-level target sequence.
- SAMA(w/o pred): SAMA(w/o pred) is a degraded model of SAMA that removes the process of skill prediction for the ablation test.
- SAMA(w/o graph): SAMA(w/o graph) is another degraded model of SAMA that removes the process of skill refinement.
4.1.3 Network Configuration
In all models, we pretrain word2vec (Mikolov et al., 2013) in the job posting dataset. We set the word embedding dimension as 100 and the hidden vector size as 400 in both encoding and decoding. We set
$^{5}$ https://github.com/Hit-SCIR/pyltp
3The skill labels are silver standard, because it was not created by an expert but extracted by a trained model. 4The job requirement text is gold standard, because it was written by human and put out online.
Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-3	ROUGE-4
S2SA	44.78	29.96	20.33	13.11	44.43	20.02	8.87	3.62
DelNet	37.10	25.35	18.28	12.62	44.21	19.29	8.42	3.08
VPN	34.15	23.26	16.90	11.68	40.16	16.82	6.96	2.63
SAMA(w/o pred)	44.70	31.59	23.09	16.32	45.87	22.78	11.25	5.75
SAMA(w/o graph)	45.49	31.89	23.32	16.40	45.93	22.85	11.37	5.84
SAMA	46.15	32.44	23.77	16.83	46.37	23.27	12.17	6.16
Table 2: Word overlap based metrics.
the maximum number of words in each sequence of skills and each job requirement as 30 and 150, respectively. Also, the weighted parameters  $\lambda$  and  $\mu$  are set as 0.5 and 1.4, respectively. The threshold  $\theta$  is set as 100. We apply dropout (Zaremba et al., 2014) at a rate of 0.3. Models are trained for 15 epochs with the Adam optimizer (Kingma and Ba, 2015), and the batch size is 5.
4.1.4 Evaluation Metrics
To evaluate the performance of SAMA, we employ the following metrics:
Word overlap based metrics: To evaluate the overall text generation quality, we employ BLEU-N (Papineni et al., 2002) and ROUGE-N (Lin, 2004) as evaluation metrics, in which BLEU-N is a kind of precision-based metric and ROUGE-N is a kind of recall-based metric.
Skill prediction metrics: Since the correctness of generated skills is of great importance in JPG, we further evaluate the quality of skills in generated job requirements, using Precision, Recall, and F1 value. To achieve this, we extract skills in the ground truth and generated text by a matching method based on the skill vocabulary  $V_{skill}$ .
Human-based evaluation: Since it is difficult to measure the comprehensive quality of the generated texts, i.e., both fluency of the texts and accuracy of the skills, in addition to automatic metrics above, we conduct a subjective evaluation following. Three graduated student volunteers are asked to evaluate the generated paragraphs. We randomly sample 50 pieces of data from the testing set. The job requirements generated by different models are pooled and randomly shuffled for each volunteer. Each generated paragraph is evaluated as bad (irrelevant skills or disfluent sentence), normal (basic relevant skills and fluent sentence), or good (rich and relevant skills and fluent sentence).
Table 2 shows the results of word overlap based metrics. In terms of BLEU-N and ROUGE-N,
Figure 4: Skill prediction metrics.
SAMA performs the best in all word overlap based metrics, which suggests that our model obtains more overlapped words with the ground truth. SAMA(w/o graph) and SAMA(w/o pred) obtain competitive results, and both are significantly better than baselines, which demonstrates the effectiveness of skill prediction and prior knowledge of skills, respectively.