<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_2.png" /></figure>

# Employing Natural Language Processing
Techniques for Online Job Vacancies
Classification

George Varelas¹(✉), Dimitris Lagios¹, Spyros Ntouroukis¹, Panagiotis Zervas¹, Kenia Parsons², and Giannis Tzimas¹

¹ Data and Media Laboratory, Department of Electrical and Computer Engineering, University of Peloponnese, Patras, Greece

g.varelas@go.uop.gr, {d.lagios,douroukis,p.zervas,tzimas}@uop.gr

² The World Bank, Washington, DC, USA

kparsons@worldbank.org

**Abstract.** With the advances in natural language processing and big data analytics, the labor market community has introduced the emerging field of Labor Market Intelligence (LMI). This field aims to design and utilize Artificial Intelligence (AI) algorithms and frameworks to analyze data related to the labor market information for supporting policy and decision-making. This paper elaborates on the automatic classification of free-text Web job vacancies on a standard taxonomy of occupations. In achieving this, we draw on well-established approaches for extracting textual features, which subsequently are employed for training machine learning algorithms. The training and evaluation of our machine learning models were performed with data extracted from online sources, pre-processed, and hand-annotated following the ISCO taxonomy. The results showed that the proposed model is very promising. The advantage is its simplicity. After its application to a relatively small and difficult to clean dataset, it achieved a good accuracy. Furthermore, in this paper we discuss how real-life applications for skill anticipation and matching could benefit from our approach.

**Keywords:** Natural language processing · Labor market · ISCO taxonomy · prediction

## 1 Introduction

In a short period of time, the Internet has evolved into a powerful platform that has dramatically altered the way people do business and communicate. Even from the advent of the Internet, labor market stakeholders (including the employers and the job seekers) have used various online methods to advertise and search for jobs. In turn, this has led to the accumulation of a huge amount of data related to the labor market and the skills required. All these have resulted in researchers identifying the Web as a research platform and a data source, pointing out its value for labor market analysis.
Due to its nature as a data source, the Web contains a plethora of diverse digital sources available for extracting various kinds of information. This diversity imposes the need for the efficient identification of quality data sources. Various sources may be appropriate for extracting trends related to employer demands, job seekers' interests, and hard and soft-skills identification. Some sources can be considered as job-focused (i.e., EURES, National Public Employment Services Portals, and private sector online Job portals). In contrast, others can have generic content (i.e., Twitter user interaction data, YouTube post metadata and content, LinkedIn, etc.) or may have subjective content related to labor markets such as portals that hold public administrative microdata (i.e., ministries or public organizations reports), national statistical office microdata (i.e., data relevant to skills from statistical surveys, census of the population) and large-scale skills survey and skills measurement microdata (i.e., public and private organizations beyond national statistical offices undertake substantial enterprise skills surveys, CV analysis, etc.).

For extracting meta-information from textual data, natural language processing approaches have been explored and utilized. To this end, the term “Labor Market Intel- ligence” (LMI), is referring to the use and design of AI algorithms and frameworks for Labor Market Data to support decision-making.

In this paper, we describe how the problem of classifying and extracting practical knowledge from Web Labor Market Data has been addressed and formulated as a Text Classification problem. More specifically, our research contributes to addressing the problem of classifying textual data from online job vacancies in the ISCO-08 taxonomy (ISCO stands for International Standard Classification of Occupations). The remainder of this paper is structured as follows: Sect. 2 reviews in detail prior work in the areas of job vacancies data collection from web sources as well as machine learning approaches applied for text classification. In Sect. 3, data collection schemes and preparation methods are described. The methodology of our approach is presented in Sect. 4, while Sect. 5 discusses the implementation and the results. Finally, the concluding remarks of the paper are summarized in Sect. 6.

## 2 Related Work

Back in 1957, H. P. Luhn [1] published an article in which the foundations of NLP had been set. In this article, he was suggesting automatic system that would be able to classify documents according to a given topic. Nowadays, Natural Language Processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence (AI) aiming at giving machines the ability to read, understand, and deliver meaning. There is a vast utilization of NLP approaches in various domains such as finance, media, and human resource. The NLP system needs to understand text, sign, and semantics properly. Since more than eighty percent of data about an entity are available only in unstructured form, many methods have been proposed for text and symbol understanding in an NLP workflow. Such methods are speech reorganization, probabilistic language models, word embedding, and text classification. Text clarification is the process of categorizing the text into a group of words. Using NLP, text classification can automatically analyze text and then assign a set of predefined tags or categories based on its context. Overall, NLP is used for sentiment analysis, topic detection, and language detection, among others.
Recently, text classification methods have been widely studied in extracting knowl-
edge from unstructured online job postings. Job recommendation systems heavily depend
on robust text classification modules for extracting knowledge related to the occupation
code [2] of a job vacancy. In their work, Lu et.al [3] presented a recommender system for
job seeking and recruiting employing graph modeling. Shalaby et.al in [4] have proposed
a homogeneous Graph-Based architecture for job recommendation tasks. In [5] and [6]
Boselli and et.al utilized machine learning approaches related to text classification for
job web vacancies classification and other labor market intelligence tasks. Finally, in [7]
the potential to assign occupations to job titles contained in administrative data using
automated, machine-learning approaches was explored.

Bethmann et.al [8] applied Naive Bayes and Bayesian Multinomial to predict occu-
pations from open-ended survey questions on respondents. They estimated correctness
probabilities for every occupation category given the information in the training data. As
labels, they used KldB2010 which is the German occupation code. Their results showed
that in order to classify text relative to occupation correctly, one needs a large sample of
quality data.

Russ Daniel, et.al [9] used Computer-based coding of free-text job descriptions to identify occupations in epidemiological studies efficiently. They used an ensemble classifier that automatically assigned SOC codes based on free-text job titles, free-text tasks, and SIC-1987. In external comparisons of jobs obtained from a case-control study, the classifier assignments had an overall agreement with expert coders that ranged from 45% at the 6-digit level to 76% at the 2-digit level.

Mukherjee et.al [10] compared the results of several algorithms to determine Stan-
dard Occupational Classification Codes from Job Descriptions in Immigration Petitions.
They showed that TF-IDF n-gram (Term Frequency-Inverse Document Frequency) based
support vector classifier with radial basis function (SVC-RBF) achieves the highest clas-
sification accuracy of 81%. However, they found also that doc2vec based random forest
achieves the highest precision score. Their results are in agreement with our findings.

Adomavicius and Tuzhilin in [11] examine possible extensions. Content-Based meth-
ods read a user profile, extract a set of features, and determine if an item is appropriate
for this profile as a recommendation. TF-IDF and clustering are common algorithms
for this approach. In Hybrid methods of recommendation, different methods can be
combined. For instance, one can implement separate collaborative and content-based
systems. Then, the outputs obtained from individual recommender systems can be com-
bined into one final recommendation using either a linear combination of ratings or a
voting scheme. This approach was used in our paper.

Another article written by Amato, et.al [12] examined a different point of view. They exploited the affinity relation that the ISTAT occupation classifier (Italian standard classifier for Occupations) defines for some occupations. The rationale of this relationship is to express that a professional profile (e.g., programmer) is akin to another one, such as a software developer.
# 3 Corpora Pre-processing and Annotation

Having a set of four Greek online portals, web crawling was carried out, and a vast amount of content was gathered. The description of the overall process is schematically described in Fig. 1. The result of the initial procedure was a dataset having a considerable noise. With the term noise, we refer to objects that are irrelevant to the project's scope, such as city names. So, Data Cleansing was necessary in order to improve the results of the model. In this procedure, we removed characters like "*...etc.". and words the stop words (Example of stop words "mustn't, also, although etc.". From each vacancy, only 2 fields were kept: the title and the description of the job posting.

After the data collection and the data cleansing processes, the dataset was split into smaller sets and each one was assigned to an expert to label it with the correct occupation code manually. In our case, the job title concatenated with the job description was used as input text and the accuracy of the result, which is the ISCO occupation code, was measured. This procedure is the Document Classification.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Data preparation procedure</figcaption></figure>

As a next step, we performed a lemmatization procedure that removes suffixes and leaves words in their root form.

Stemming was not used because it would not contribute considerably, and this way, we ended up in a simpler model. In the end, our resulting dataset contained 6,902 ads.

## 3.1 Exploratory Analysis

The distribution of Occupation Codes in our dataset is illustrated in Fig. 2. There are 113 different Codes. It is evident that the dataset is imbalanced. Greece is a tourist attraction, and subsequently, there is high demand for employees in this sector. The Occupation Code with the highest demand was that of a "Shop Sales Assistant". During the last decade, Greece has suffered an economic crisis. Many employers have chosen to work their shops by themselves (one should keep in mind that the majority of businesses in
Greece are family businesses or small companies). The job ads were collected during the reorganization of the Greek economy in which many measures to reduce unemployment were introduced.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Occupation Code label count in the dataset</figcaption></figure>

The average number of words per Occupation Code is depicted in Fig. 3. That was a problem because it adds noise (irrelevant words) to the data. Additionally, it poses problems to all the algorithms involved.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Average number of words per Occupation Code</figcaption></figure>
Cleaning this noise is one of the most challenging tasks in NLP. The irrelevant text was scattered throughout the job ads. There were many ads that had low levels of noise. After a lot of experimentation with the dataset the research team decided to keep the first 500 words (after document cleansing) of the combined title and description text data.

Typically, the job posting title has enough information to extract the type of occupa-
tion. But this is not always the case, as Amato et al. [13], and Marrara et al. [14] point
out. The job title might be the same for different but similar occupations. The details
of the occupation exist in the job description text. That is why the description was not
excluded from the analysis.

The top 100 most common words of the job postings in the form of a WordCloud
are depicted in Fig. 4, while the frequency of the 25 most common words in the form of
bar plot is depicted in Fig. 5.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Top 100 most common words</figcaption></figure>

Although Figs. 4 and 5 show the same thing (topmost common words), they display
it from different perspectives. Figure 4 is more qualitative, showing which words are
there, and Fig. 5 is more quantitative.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Frequency of 25 most common words</figcaption></figure>

# 4 Methodology

As mentioned in previous sections, a classifier aims to use a dataset of pre-classified documents for training a model that will be able to classify data having not seen yet. For implementing our text classification functionality, we followed the steps depicted in Fig. 6.

The first step in training a text classifier is that of feature extraction. By feature extraction, we refer to the task of representing a given text in a form that is suitable for applying it as input to a machine learning algorithm. A method that has been utilized successfully in NLP tasks is that of word-embedding. Although word-embedding can be considered as a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc., on a large corpus). Word embedding techniques help extract information from the pattern and occurrence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.

## 4.1 Feature Extraction

In our research for forming the feature vector of our dataset, we utilized Doc2Vec Word Embedding algorithm.
To explain what Doc2Vec is, we must start from the Word2Vec [15]. The output of this algorithm is a numeric representation of each word in a document which is able to capture words with a similar context. It uses two methods to obtain this. Skip Gram and Common Bag Of Words (CBOW). The CBOW method considers each word's context (a surrounding window around the word under consideration) as the input. It has as output (prediction) a word corresponding to this context. The skip Gram is the opposite of CBOW. Instead of predicting one word each time, it uses one word to predict all surrounding words ("context").

Doc2Vec has many similarities with Word2Vec, but the goal remains the same. To represent a document numerically as a vector but without considering its length. It borrows the Word2Vec vectors and adds one more. The Paragraph Vector. It is actually a small extension of the CBOW (Continues Bag of Words) algorithm used in Word2Vec. At the end of the training, the document vector D holds a numeric representation of the document. This model is called Distributed Memory version of Paragraph Vector (PV-DM). It acts as a memory that remembers what is missing from the current context or as the topic of the paragraph. While the word vectors represent the concept of a word, the document vector intends to represent the concept of a document. Another algorithm, similar to Skip-Gram may be used. It is called Distributed Bag of Words version of Paragraph Vector (PV-DBOW), and it is the equivalent of CBOW from Word2Vec. We utilized this algorithm in our analysis because it is faster and has lower needs in terms of memory, since the word vectors are not saved.

## 4.2 Methods

The SVM Linear, Random Forests, KNN, SGD (Stochastic Gradient Descent), and MLP Neural Network classifiers were implemented. These methods were used as the baseline for our Hard Voting ensembling model.

<figure><img src="image_8.png" /><figcaption>Fig. 6. (a) Training of text classification model, (b) Trained model utilization for prediction</figcaption></figure>
In SVM we used Linear kernel with regularization equal to 10, in Random Forest we used 600 estimators, in MLP we used 512 units in 1 hidden layer with “Relu” activation and “Adam” solver, in KNN we took 10 neighbors depending on distance weights with Euclidean distance, and in SGD Classifier we used “modified_huber” as loss function and “L2” penalty. All these parameters came after experimentation.

## 4.3 Evaluation Metrics

The effectiveness of the classifiers has been evaluated based on their accuracy and the F1-score. Since the data are highly imbalanced, the F1 score is more appropriate as an index. The F1 score is the harmonic mean of precision and recall of the model. Its lowest value is 0, which means that no classification is accomplished with the model under consideration, and the highest value is 1, which means that the perfect classification has occurred. The weighted average of the precision of each class is calculated for this multiclass task.

Precision's meaning is: Within everything that has been predicted as a positive, precision counts the correct percentage.

Recall's meaning is: Within everything that actually is positive, how many did the model succeed to find.

$$Accuracy = \frac{true positives + true negatives}{total predictions}$$

$$Precision = \frac{true positives}{true positives + false positives}$$

$$Recall = \frac{true positives}{true positives + false negatives}$$

F1 score combines these two metrics into one and works well in imbalanced data. The F score is defined by the relationship:

$$F_{\beta} = (1 + \beta^2) \times \frac{precision \times recall}{(\beta^2 \times precision) + recall}$$

The β is a factor that indicates the importance of recall over precision. Here the β is set equal to 1 because the importance of precision is equal to the importance of recall.

# 5 Experiments and Results

The labeled dataset was randomly partitioned into training and test sets. The partition was performed to split the vacancies as: 80% in the training set and 20% in the test set (Table 1).
<table><caption>**Table 1.** Results of individual algorithms</caption><thead><tr><td>Model</td><td>Test accuracy results</td><td>CV accuracy results</td><td>CV F1 score results</td></tr></thead><tbody><tr><td>Random Forest</td><td>79%</td><td>75.69% (±0.63)</td><td>74.83% (±0.64)</td></tr><tr><td>MLP</td><td>85.69%</td><td>83.83% (±0.50)</td><td>83.27% (±0.51)</td></tr><tr><td>Knn</td><td>74.22%</td><td>70.39% (±1.29)</td><td>69.37% (±1.28)</td></tr><tr><td>SVM</td><td>82.91%</td><td>80.04% (±1.22)</td><td>79.99% (±1.16)</td></tr><tr><td>SGD</td><td>85.95%</td><td>83.39% (±0.37)</td><td>83.54% (±0.29)</td></tr></tbody></table>

Having the above results, we implemented an ensemble method. More specifically, we implemented a Hard Voting algorithm. The results were better than the ones above (Table 2).

<table><caption>**Table 2.** Results of the ensemble method</caption><thead><tr><td>Model</td><td>Test accuracy results</td><td>CV accuracy results</td><td>CV F1 score results</td></tr></thead><tbody><tr><td>Hard voting</td><td>88.85%</td><td>88.04% (±0.57)</td><td>87.9%</td></tr></tbody></table>

Moving one step further, we calculated the test accuracy on 3 and 2 digits of the occupation code (Table 3).

<table><caption>**Table 3.** The Sub-major and minor class of Occupation Code accuracy</caption><thead><tr><td>Model</td><td>(3 digits Minor class)</td><td>(2 digits Sub-major Class)</td></tr></thead><tbody><tr><td>Hard voting</td><td>89.93%</td><td>90.95%</td></tr></tbody></table>

The ensembling model of hard voting performed better than any other tested model, although some algorithms like MLP, SVD, and SGD performed so well that could be considered as standalone models that need better hyperparameter tuning and a more extensive training dataset.

# 6 Discussion

This problem of assessing the Occupation Code from job posting has singularities. It is highly dependent on the country and the distribution of different occupations in these postings. So, it results in different measurements of accuracy in modeling, and this is why one can find different accuracies in the literature.

In publication [5] the score of accuracy was 90.7%. During their scraping phase, they gathered 2,295,603 job vacancies. This a number large enough to help in the confidentiality of the results. One advantage these researchers had was that these job postings were written in English.
In this paper we presented an ensembling hard voting algorithm for the classification of job postings according to theISCO Occupation Codes. The data was collected from job posting from Greek portals. The number of job postings in our case was not very extensive, but we are confident that our model would achieve higher accuracy if we had a larger volume of postings, this being the next step in our research.

Additionally, the model achieves high accuracy in the Minor and sub-major groups ofISCO-08. For example, it classifies a “Legislator” correctly under the “Legislators and senior officials” minor group with a probability of 89.93% and under the “Chief executive, senior official and legislator” sub-major group with a probability of 90.95%.

Nowadays, Deep Learning techniques are used in the vast majority of NLP tasks. This paper proves that simpler methods if they are combined properly can be equal or more powerful to deep learning. The advantage is that it can be computationally more efficient.

Checking our results under the context of another country and language is also the natural continuation of our research. It will be interesting to examine the performance of the model in other languages since most of the research in this sector is confined to the English language.

## References

1. Luhn, H.P.: A statistical approach to mechanized encoding and searching of literary information. IBM J. Res. Dev. **1**, 309–317 (1957). https://doi.org/10.1147/rd.14.0309

2. International Labour Office, International Standard Classification of Occupations, International Labour Organization (2012)

3. Lu, Y., Helou, S. E., Gillet, D.: A recommender system for job seeking and recruiting website. In: Proceedings of the 22nd International Conference on World Wide (2013). https://doi.org/10.1145/2487788.2488092

4. Shalaby, W., et al.: Help me find a job: a graph-based approach for job recommendation at scale. In: IEEE International Conference on Big Data (BIGDATA) (2017). https://doi.org/10.1109/BigData.2017.8258088

5. Boselli, R., et al.: WoLMIS: a labor market intelligence system for classifying web job vacancies. J. Intell. Inf. Syst. **51**(3), 477–502 (2017). https://doi.org/10.1007/s10844-017-0488-x

6. Boselli, R., Cesarini, M., Mercorio, F., Mezzanzanica, M.: Using machine learning for labour market intelligence. In: Altun, Y., et al. (eds.) ECML PKDD 2017. LNCS (LNAI), vol. 10536, pp. 330–342. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71273-4_27

7. Ikudo, A., Lane, J., Staudt, J., Weinberg, B.: Occupational classifications a machine learning approach. In: IZA - Institute of Labor Economics, no. 11738, August 2018, 9th International Proceedings on Proceedings, pp. 1–2 (2010)

8. Bethmann, A., Schierholzy, M., Wenzig, K., Zielonka, M.: Automatic coding of occupations. using machine learning algorithms for occupation coding in several german panel surveys. In: Extensible Public Opinion: WAPOR 67th Annual Conference, Nice (2014)

9. Russ, D.E., et al.: Computer-based coding of free-text job descriptions to efficiently identify occupations in epidemiological studies. Occup. Environ. Med. (2016). https://doi.org/10.1136/oemed-2015-103152

10. Mukherjee, S., Widmark, D., DiMascio, V., Oates, T.: determining standard occupational classification codes from job descriptions in immigration petitions (2021). arXiv:2110.00078v1, https://doi.org/10.48550/arXiv.2110.00078
11. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. **17**(6), 734–749 (2005). https://doi.org/10.1109/TKDE.2005.99

12. Amato, F., Boselli, R., Cesarini, M., Mercorio, F., Mezzanzanica, M., et al: Classification of web job advertisements: a case study. In: SEBD (2015)

13. Amato, F., Boselli, R., Cesarini, M., Mercorio, F., Mezzanzanica, M., et al: Challenge: processing web texts for classifying job offers. In: Proceedings of the 2015 IEEE 9th International Conference on Semantic Computing (IEEE ICSC 2015), pp. 460–463 (2015). https://doi.org/10.1109/ICOSC.2015.7050852

14. Marrara, S., et al.: A language modelling approach for discovering novel labour market occupations from the web. In: International Conference on Web Intelligence (2017). https://doi.org/10.1145/3106426.3109035

15. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: International Conference on Learning Representations (2013). https://doi.org/10.48550/arXiv.1301.3781