Inferring job vacancies from online job advertisements : 2021 edition.
Date: 2021-01-26
Institutions: Eurostat
Subjects: big data, electronic government, employment service, employment statistics, job vacancy, labour market

Inferring job vacancies
from online job
advertisements

MACIEJ BERĘSEWICZ AND ROBERT PATER

2021 edition

WORKING

STATISTICAL
PAPERS

Inferring job vacancies
from online job
advertisements

MACIEJ BERĘSEWICZ AND ROBERT PATER

2021 edition

Manuscript completed in January 2021  

The Commission is not liable for any consequence stemming from the reuse of this publication. 

Luxembourg: Publications Office of the European Union, 2021 

© European Union, 2021 

The reuse policy of European Commission documents is implemented based on Commission Decision 
2011/833/EU of 12 December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 
39). Except otherwise noted, the reuse of this document is authorised under a Creative Commons 
Attribution 4.0 International (CC-BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This 
means that reuse is allowed provided appropriate credit is given and any changes are indicated. 

For any use or reproduction of elements that are not owned by the European Union, permission may 
need to be sought directly from the respective rightholders. The European Union does not own the 
copyright in relation to the following elements: 
For more information, please consult: https://ec.europa.eu/eurostat/about/policies/copyright 

Copyright for the photograph: Cover © LeoWolfert/Shutterstock 

Theme: Population and social conditions 
Collection: Statistical working papers 

PDF: 

ISBN 978-92-76-23299-5 

ISSN 2315-0807 

doi: 10.2785/963837 

KS-TC-20-008-EN-N 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Abstract

Abstract

Online data creates possibilities as a supplement to official statistics. With the development and popu-
larity of online job boards, one of such big data are job advertisements. Online job advertisements are
an example of data that may support and deepen labour market statistics. At least two applications of
such data are especially promising: (i) job offers as a leading indicator of the labour market situation,
(ii) a source of additional structural and qualitative information at a detailed level, for example on skills.
However, online data come from non-probability samples. Before supplementing official statistics prob-
lems of coverage and representativeness of online job offers need to be addressed.

This study analyses data produced by Cedefop in a pan-European approach, based on online job adver-
tisements (OJA) that it has collected and from which it has extracted several statistical variables. Cedefop
supplies the largest database of OJA data in Europe and involves representatives of each of the European
Union Member-States. Such a rich database may possibly supplement job vacancy statistics, obtained
with probability-based surveys conducted at national level.

The main objective of the study was to develop estimator(s) for the number of job vacancies from OJA
data on online job advertisements, accounting for the differences in the statistical unit and coverage. We
start with defining the relation between a job advertisement and a job vacancy, and review approaches
to analysis of online job offers presented in economic literature. Next, we describe potential statistical
methods that may be used to infer job vacancies from job ads. We start our empirical analysis with
comparison of historical data on both statistics. Then we proceed with the application of presented
methods to Cedefop and Eurostat datasets.

We find that although Cedefop’s OJA data are promising, at the present stage they are still experimental
and could not be used to estimate the total number of vacancies. Both the time trends and the structure
of job advertisements according to country, industry and occupation significantly differ from the ones
of job vacancies. However, we do find promising results for certain countries and industries. We also
explain the considerable limitations of our current analysis that come from: unknown quality of some of
the procedures followed to produce the dataset, short time series of available data, and lack of unitary
data from the surveys of job vacancies. We recommend taking a subsample of Cedefop’s data that will
be suitable to predict job vacancies and redesigning the study that will cover certain disadvantages of
the current approach. We make a number of detailed suggestions, among which: (i) conduct an audit
sample, (ii) reconsider the procedure of choosing sources for online job offers, focusing on obtaining
more specified information, (iii) link the final information on the estimated job offer with the source of
job advertisement and provide measures of the quality of machine learning methods in order to be able
to apply proper inference methods, (iv) collect information about a company advertising a job and link
it with the business register, (v) use Cedefop’s data as source of additional qualitative information about
job vacancies.

Keywords:

job vacancies, online job advertisements, big data, non-probability samples.

Authors: Maciej Beręsewicz (1) , Robert Pater (2) .

Acknowledgement: On behalf of Eurostat - Unit B1 the study was coordinated by Mátyás Mészáros
and Fernando Reis, with contributions from Raquel Paulino. The authors would like to thank to all con-
tributors from Eurostat and Cedefop as well to Fabio Bacchini (external collaborator) for their helpful
comments.

The study was implemented by GOPA, as contractor of Eurostat for the framework contract on Method-
ological Support (ref. 2018.0086). Project management was ensured by Florabela Carausu.

(1) Poznań University of Economics and Business, maciej.beresewicz@ue.poznan.pl; Statistical Office in Poznań, Poland,

m.beresewicz@stat.gov.pl

(2) University of Information Technology and Management in Rzeszów, rpater@wsiz.rzeszow.pl; Educational Research Institute in

Warsaw, Poland, r.pater@ibe.edu.pl

Inferring job vacancies from online job advertisements

3

Table of contents

Table of contents

Abstract................................................................................................................... 3

1 Introduction ....................................................................................................... 5

2 General quality issues ......................................................................................... 7
2.1 Online Job Advertisements, Vacancies and Capacity Utilization..............................
7
2.2 Using online job advertisements for economic research ........................................ 10

3 Methods ............................................................................................................. 14
3.1 Basic settings .................................................................................................. 14
3.2 Unit-level methods........................................................................................... 15
3.2.1 Modelling misclassification ...................................................................................... 15
3.2.2 Non-ignorable selection........................................................................................... 16
3.2.3
Single source capture-recapture .............................................................................. 17
3.2.4 Multiple source capture-recapture........................................................................... 19
3.3 Domain-level methods ..................................................................................... 22
3.3.1 Measurement error models ...................................................................................... 22
3.3.2 Modelling of under-reporting .................................................................................. 23
3.4 Summary ........................................................................................................ 26

4 Comparing OJA data with job vacancy statistics ................................................... 27
4.1 Aligning job advertisements data to Job Vacancies Statistics ................................. 27
4.2 Relation between online job offers and job vacancies ........................................... 32
4.2.1 Descriptive statistics................................................................................................. 32
Cross-sectional regressions ...................................................................................... 37
4.2.2
Panel data regressions.............................................................................................. 39
4.2.3

5 Results ................................................................................................................ 45
5.1 Methods and their assumptions ......................................................................... 45
5.2 Naïve trimming................................................................................................ 47
5.3 Zero-one truncated capture-recapture results...................................................... 49
Estimation procedure............................................................................................... 49
Application............................................................................................................... 50
Countries with under 200k advertisements.............................................................. 51
Countries over 200k advertisements ........................................................................ 53
5.4 Conclusions .................................................................................................... 55

5.3.1
5.3.2
5.3.3
5.3.4

6.1.1
6.1.2

6 Findings and recommendations........................................................................... 56
6.1 Main findings .................................................................................................. 56
Correlation between OJA and JVS ............................................................................ 56
Population size estimation ....................................................................................... 56
6.2 Main limitations .............................................................................................. 57
6.3 Recommendations ........................................................................................... 58
6.3.1
For the OJA dataset .................................................................................................. 58
6.3.2 Data sources............................................................................................................. 58
Audit sample ............................................................................................................ 58
6.3.3
Additional scraping .................................................................................................. 59
6.3.4
6.4 Conclusion ...................................................................................................... 60

7 References .......................................................................................................... 61

References ............................................................................................................... 61

Annex...................................................................................................................... 64

Inferring job vacancies from online job advertisements

4

Introduction

1

1 Introduction

The literature on the use of online job offers has been thriving in the latest years. Online job offers may
provide much detail about behaviour of economic entities. However, the literature and methods are still
at the early stage of fast development. They especially need statistical accuracy. So far, researchers paid
special attention to obtain similar distributions of online data with data from probability-based surveys
for certain variables. The literature still needs to develop an approach to reduce representation bias
observed in online job offers.

The largest collection of online job offers for analytical purposes in Europe is conducted by the Euro-
pean Centre for the Development of Vocational Training (Cedefop). This pan-European approach applies
methods to collect online job advertisements in all European Union member states. Cedefop’s system of
collecting online job offers provides experimental data on online job advertisements (OJA), as well as ex-
perimental data on skills’ demand (Skills Online Vacancy Analysis Tool for Europe (Skills OVATE)(3) ). The
former is the subject of this study.

Experimental data on online job advertisements provided by Cedefop are the result of the coopera-
tion between various institutions from the European Union Member-States within the project ”Real-time
labour market information on skill requirements: feasibility study and working prototype”. The advan-
tage of the data is not only their European scope, but also their continuous collection. This provides the
opportunity to track market trends and possibly support official statistics in monitoring job vacancies.

In order to collect big data on online job offers, Cedefop uses extensive cooperation with national ex-
perts, institutions providing data on job advertisements, as well as webscraping algorithms, download-
ing publicly available data from online job boards. The landscaping of online job advertisements is wide
(Cedefop, 2019b). At the stage of writing this report, results from Cedefop’s research were available for
the period 2018Q2-2020Q1.

Cedefop uses dedicated ontologies created to collect, transform, and classify job advertisements (Cede-
fop, 2019c). Those methods are based on artificial intelligence techniques (Colombo et al., 2019). Thanks
to them Cedefop is able to generate a very detailed data. These data include more variables than probability-
based surveys can possibly contain. These include mainly:

• time

• area

• occupation

• skill

• contract

• education

(3) See https://www.cedefop.europa.eu/en/data-visualisations/skills-online-vacancies

Inferring job vacancies from online job advertisements

5

Introduction

1

• industry

• source.

Thanks to this approach, the obtained job offers estimates can be controlled for some pitfalls of online
data. For example, the ”type of contract” indicates whether an advertisement concerns a work contract
or an internship/traineeship. This can be used to properly calculate the number of job positions.

However, it needs to be underlined that at this stage, methods to collect and analyse online big data are
still developing and are not unerring. As Cedefop underlines, ontologies developed to organise collected
data are still imperfect, and are subject to continuous correction. Also, even though Cedefop gathers
data from many sources, OJA represent only part of the vacancy market (Cedefop, 2019a).

The objective of this study is to develop estimator(s) for the number of job vacancies from data on online
job advertisements, accounting for the differences in the statistical unit and coverage.

In order to fulfil the general objective of the study, we first review the literature on online job offers, and
review methods that can be used to infer job vacancies from job offers. The theoretical part is based on
economic and statistical literature. In the empirical part we compare online job offers and vacancies. We
suggest a method for inference job vacancies from Cedefop’s online job offers, and present limitations of
such a study. Finally, we apply the chosen method, calculate its error and formulate recommendations
for further studies.

In Chapter 2 we identify the relationship between the populations of the two objects of our study, “online
job advertisement” and “job vacancy”. We point various sources of potential bias in job advertisements
used as proxies of job vacancies. Then we review economic literature on the approach to collecting,
transforming, and analysing online job ads.

In Chapter 3, we investigate possible inference methods that can be applied to estimate job vacancies
from online job advertisements data. We assess the available auxiliary information. We distinguish those
methods that can be applied with the existing auxiliary data.

In Chapter 4, we compare online job advertisement (OJA) data with job vacancies statistics (JVS), by
analysing the relations between Cedefop’s OJA dataset with Eurostat’s JVS. We start by aligning both
datasets, discuss data limitations and merging approach. Then we analyse them, using a general to
specific procedure. We consider available breakdowns: time in quarters, countries, NACE sections (in-
dustries), and ISCO major occupation groups. Finally, we calculate aggregate statistics and regressions,
and we disaggregate them to individual cross-sections.

In Chapter 5, we perform the estimation of the number of job vacancies using the method selected in
Chapter 3. We also estimate the accuracy of the estimations of the number of job vacancies based on
OJA data.

The 6th chapter contains conclusions and recommendations for Cedefop and Eurostat.

Inferring job vacancies from online job advertisements

6

General quality issues

2

2 General quality issues

2.1 Online Job Advertisements, Vacancies and Ca-

pacity Utilization

A company may always have non-utilised capacity. Capacity utilisation rate is the share of potential out-
put a company could produce if all installed equipment (capital owned by a company) was fully used.
Economic theory points out that it happens when the average production cost function is at its mini-
mum. Most of the time, (physical) capital is not fully used and therefore capacity utilization is not full.
This means that there are workplaces that are not filled in the company. Even holding capital constant,
i.e.
in the short run from the economic point of view, most companies may still hire new workers to
increase output.

However, non-full capacity utilisation does not mean that there are job vacancies. A vacancy is an un-
occupied workplace that a company wants to have filled. Company representatives must take action to
hire a worker. This makes vacancy statistics highly susceptible to economic expectations. It also con-
nects vacancies to the short-run economics. That is why vacancies are generally strongly pro-cyclical,
and exhibit a weak trend. A vacancy does not explicitly refer to the long run, because a company does
not publish a notice that will hire a worker after certain investments increase the firm’s capital. It takes a
long time for investments made by a company to be fully reflected in terms of job vacancies.

On one side, we have economic expectations, which are imaginary, even though they are based on real
factors. On the other side, we have employment, which is well documented for legal and other reasons.
Job vacancies fall in between these two categories. Also, in some companies employment decisions are
formalized and centralized while in others they are not, which poses a measurement problem (Dunlop,
1966). Figure 1 shows the theoretical relation between non-utilized capacity, job vacancies and online
job advertisements (i.e. job offers)(4) .

Job openings can refer to one of two types of data: 1) newly created jobs across a certain period, for
example a month, that is, flow of job openings, or 2) vacancies in the end of a given period, that is, stock
of job openings.

Eurostat(5) defines a job vacancy as a paid post that is newly created, unoccupied, or about to become
vacant under two conditions:

1. employer is taking active steps and is prepared to take further steps to find a suitable candidate for a

job from outside the enterprise concerned; and

(4) A ”Job advertisement” is a notice containing a ”job offer”. Online job advertisement is placed on an Internet website with online
job offers (commonly referred to as an ”online job board”). Because of this, we use terms ”job advertisements” and ”job offers”
as synonyms. However, for consistency purposes, throughout the report we generally use the term ”job advertisements” (JA),
and for their Internet portion the term ”online job advertisements” (OJA)

(5) See https://ec.europa.eu/eurostat/cache/metadata/en/jvs_esms.htm

Inferring job vacancies from online job advertisements

7

General quality issues

2

Figure 1: Non-utilised capacity, vacancies and online job offers

2. employer intends to fill the job position either immediately or within a specific period.

Eurostat also clarifies what ”Active steps to find a suitable candidate” means by specifying the following
activities:

• notifying the job vacancy to the public employment services,

• contacting a private employment agency/headhunters,

• advertising the vacancy in the media (for example internet, newspapers, magazines),

• advertising the vacancy on a public notice board,

• approaching, interviewing or selecting possible candidates/potential recruits directly,

• approaching employees and/or personal contacts,

• using internships.

The first part of the first condition is fulfilled by an online job advertisement, since posting one might
be considered an ”active step” to find a candidate for a job. For the online job advertisements we do
not know whether the second part of the first condition is fulfilled. Enterprises might or might not be
prepared to ”take further steps”, e.g. to make an interview with potential candidates for the job. While in
the case of paid advertisements, a company is most likely willing to take further steps to employ a person,
it might not always be true for free-of-charge postings. By posting a job advertisement, a company might
potentially only try to investigate the market – the number and potential skills of job candidates (this is
sometimes referred to as ”ghost vacancies”).

It is likely that by using
Likewise, we cannot be certain about the first part of the second condition.
paid job advertisements websites, an employer intends to ”fill the job position”, but it might not hold for
free-of-charge websites. This means that it is important, when using OJA data, to differentiate between
these types of websites. We might consider the second part of the second condition fulfilled by online
job advertisements, as Eurostat does not specify the ”specific period”.

To sum up the definition stage, we can see that publishing a job advertisements is one of the meth-
ods to notify a job vacancy. Our doubts on whether vacancies fully encompass job advertisements are

Inferring job vacancies from online job advertisements

8

General quality issues

2

connected to the truthfulness of a company willingness to hire a job seeker. However, this heavily de-
pends on the credibility of a considered website with job advertisements. Credible websites stipulate
that doubtful job advertisements will be removed and list conditions when this happens. Providing that
the initial landscaping and selection of websites ensures that only credible websites are considered, we
might assume that in terms of definition, publishing online job advertisements is a valid method of no-
tifying a job vacancy.

The sources of potential bias in online job advertisements as vacancy measures are the following (types
of errors indicated in square brackets):

• job advertisements do not include the part of vacancies for which the employer is looking for an
employee using methods other than an advertisement on the Internet; for low-paid jobs it might
be an informal way or a sign in the window; also some high-paid or scarce occupations may be
underrepresented, because companies seek such workers through HR agencies, e.g. directors,
data scientists; even online job search may take different forms, not only job postings but also
job-task search or crowdwork [under-coverage error],

• online job advertisements are not considered by households and companies that do not have or

rarely use the Internet [under-coverage error],

• not all online job advertisements are collected, since not all websites are covered; it is virtually

impossible to cover all of them [under-coverage error due to selection],

• since the web data gathering algorithm does not operate all the time in a given spot on all web-
sites, not all advertisements published in each month will be considered, as they are characterized
by varied and unknown length of stay on the website [under-coverage error due to data collec-
tion],

• it is not perfectly clear whether job advertisements constitute a stock or flow; generally job offers
are gathered by a web data gathering algorithm in a specific moment, so they should constitute a
stock of vacancies; however, some websites keep job advertisements for a fixed period (it is espe-
cially the case for paid job advertisements), and they may impair the differentiation between stock
and flow measures (temporal aggregation bias); price policies and market strategies of job ads
providers can also affect the number and composition of advertised job postings [over-coverage
error],

• online job advertisements over-represent certain occupations and skills and under-represent other;
overrepresentation is generally connected to vacancies for workers with higher education and in-
dustries from the private sector, which is the opposite to job advertisements from public employ-
ment services; both sources of job advertisements are, to a high extent, supplementary to each
other [under-coverage error due to self-selection],

• several vacancies may be placed in one job advertisement, which disrupts the estimation of the
number of vacancies; it might especially be true for positions not requiring higher education [over-
coverage error, unit error],

• it is difficult to check whether job postings are fully updated; filled or withdrawn positions might
not mean that a job advertisement is withdrawn immediately and might bias their stock upwards.
This bias might last until the job advertisement is automatically removed from the websites (most
commonly, websites offer 30 days of publishing a job ad, but it is not the only option available).
Turrell et al. (2019) call it an ”aggregate outflow bias” [over-coverage error],

• online job market evolves in the long-run with technological development (logistic function-shaped
technological diffusion, see Barnichon (2010) and Pater (2017)), and over the business cycle, through
costs of posting vacancies online relative to costs of other recruitment methods (Cajner et al.,
2016), labour market tightness, and through changes in the number of required skills (Pater et al.,
2019); popularity of Internet will affect coverage of vacancies by online job advertisements, as dig-
ital divide does [under-coverage error due to selection],

Inferring job vacancies from online job advertisements

9

General quality issues

2

• even if the structure of job advertisements is corrected for ”representativeness” bias (e.g. by oc-
cupation, sector of activity etc.), skills requirements in job advertisements are likely to be accu-
rately specified in advertisements for high-paid jobs and only generally or not-at-all specified for
low-paid jobs; also many skills are implicit, or too trivial to be included in a job advertisement, for
example basic computer skills, even though they are still necessary for the job [selection error],

• ”job” advertisement placed online might not always refer to ”employment”. For example, intern-
ships and traineeships advertisements might refer to a job from employer’s perspective, while
they may not be considered ”employment” from a statistical and labour law points of view [over-
coverage error].

Online job advertisements form a specific fraction of the job vacancy market. The advantages of online
job advertisements as vacancy measures are as follows:

• it is possible to obtain very detailed information on job vacancies, e.g. in terms of skills and qual-
ifications; traditional sources, in particular questionnaire based sample-surveys, do not allow ob-
taining such detailed information,

• the employers are interested in placing their preferences in the advertisement, including those
related to new trends, which is an advantage over questionnaire based surveys, in which the em-
ployer does not benefit directly from accurately addressing the questions,

• job advertisements can potentially be examined in many cross-sections,

• job advertisements from previous periods can be re-examined, because they are saved in the
database; this allows for re-examination and re-calculation of measures based on these job adver-
tisements should methodology or classifications change; this also ensures comparability of data
over time,

• collecting job advertisements is relatively low-costly, compared to questionnaire based surveys,

• Internet is currently the main method used to publish job advertisements, and this market is still

developing,

• research can be conducted with high frequency that can be increased in the future,

• sample surveys are likely to under-represent new firms, as they are drawn from business registers
(see Davis et al. (2013) and Turrell et al. (2019)); job advertisements might be less susceptible to
this bias.

2.2 Using online job advertisements for economic

research

There is a growing collection of economic literature on the use of online job offers, with increasing at-
tention paid to selectivity. Most of the literature is based on the US economy, which has the longest tra-
dition in collecting first newspaper, and then online job offers as vacancy measures (see e.g. Abraham
and Wachter (1987), Kuhn and Skuterud (2004), Marinescu and Wolthoff (2016), Marinescu and Rathelot
(2018), Deming and Kahn (2018) and Hershbein and Kahn (2018)). The European literature that recog-
nizes the selectivity problem includes works of Colombo et al. (2019), Pater et al. (2019) and Turrell et al.
(2019).

Carnevale et al. (2014) estimate that in the US economy the share of online job offers in all vacancies
in 2014 was between 60% and 70%. The Conference Board discontinued traditional, newspaper-based
Help-Wanted Advertising Index in 2008, after having begun a Help-Wanted Online Index (HWOL) in 2005.
Ads in the HWOL index are collected in real-time from over 28,000 online job boards including tradi-
tional job boards, corporate boards, social media sites, and smaller job sites that serve niche markets and

Inferring job vacancies from online job advertisements

10

General quality issues

2

smaller geographic areas. In construction of HWOL, special attention is paid to time series properties, by
removing outliers, seasonal adjustment etc. Also, HWOL does not include online job aggregators (The
Conference Board, 2018). The Conference Board uses Wanted Analytics data to calculate Help-Wanted
Online Index. They show data disaggregated by region, sector of activity and occupation, but not by skill
requirements.

Cajner et al. (2016) state that there have been significant discrepancies between the stock of vacancies
implied by two US series, the JOLTS (Job Openings and Labor Turnover Survey) and the Conference Board
Help Wanted Online, which may be caused by changes in the price charged to employers when posting
online job vacancies.

State vacancy surveys is another source of job offers data in the US economy. However, they are con-
ducted by a limited number of states, sometimes with certain skill requirements, but cover only a few
geographic areas; such job offers are not comparable between states.

Marinescu and Wolthoff (2016), Marinescu and Rathelot (2018), Deming and Kahn (2018), and Hershbein
and Kahn (2018) analyse the US economy. They compare the distributions of online job offers they use,
or a subsample of them, to the results of probability-sample surveys and to the distributions of other
online job offers measures. For highly correlated results they assume that the non-probability sample
job offers are ”representative”. If the correlation is low, they either take a subsample of gathered job offers
or weight their data. Weights are based on the relation between vacancies obtained from probability-
sample survey and their online job offers. Turrell et al. (2018) and Turrell et al. (2019) for the UK economy
use weights in a few job breakdowns to obtain comparable distributions.

Marinescu and Wolthoff (2016) and Marinescu and Rathelot (2018) gather data on job offers from one US
website (CareerBuilder). This website is chosen because it provides many variables, it provides data for
job seekers and data about responses of job seekers to job offers as well. All vacancies are used or only
a random subsample within. Comparing online job offers they collected to the ones from a probability-
sample survey JOLTS it was stated that CareerBuilder represents 35% of the total number of vacancies
in the US economy. It was also ascertained that CareerBuilder data overrepresents following industries:
information technology, finance and insurance, and real estate, rental, and leasing, and underrepresents
state and local government, accommodation and food services, other services, and construction.

They merge three datasets extracted from CareerBuilder’s. The first one is a random sample of registered
users. Data includes residence location at the ZIP code level. The second data set is a sample of vacancies
published on the website. These vacancies are available to the job seekers. This data also contains a ZIP
code. The third dataset connects the two previous ones by showing which jobs each job seeker applied
to. An application is defined as a click on the ”Apply now” button that can be found on the full job listing
webpage.

The authors state that CareerBuilder data are ”not representative” in terms of the industry breakdown.
In terms of occupation (2-digit Standard Occupational Classification, SOC codes), the distribution of un-
employed job seekers’ occupations in CareerBuilder data are very similar to the one from CPS (Current
Population Survey, correlation of 0.71 between the shares of job seekers in each occupation in the two
datasets), and the distribution of vacancies’ occupations in the CareerBuilder data are very similar to the
distribution of vacancies in the general measure of online jobs (correlation of 0.95 with Help Wanted
Online data). Regional distribution of vacancies from CareerBuilder is very similar to vacancies from the
probability-sample survey (Job Openings and Labor Turnover Survey, correlation of 0.96 between the
shares of vacancies in each region in the two datasets). The spatial distribution of job seekers was also
analysed by comparison to the unemployed from the Current Population Survey (correlation of 0.88).

Rothwell (2014), Deming and Kahn (2018), Hershbein and Kahn (2018), and Azar et al. (2018) use an
extremely broad database of job offers. They possess microdata from nearly 100 million electronic job
postings in the United States between 2007 and 2015. Data was collected and assembled by Burning
Glass Technologies (BGT), that examined 40,000 online job boards and company websites. However, the
procedure of data gathering, cleaning, and classifying is largely unknown. They cross-validate the data
(e.g. on skills) with other measures obtained from a probability-sample survey. They weight the data

Inferring job vacancies from online job advertisements

11

General quality issues

2

by the size of the metropolitan statistical areas (MSA) labour force. BGT shows that the share of their
collected jobs online is 85% of the jobs in JOLTS in 2016.

The broad coverage of the database has a substantial strength over datasets based on a single vacancy
source, such as CareerBuilder.com, but it also has drawbacks. The drawbacks come from the fact that we
do not know the mechanism of job offer posting (e.g. paid or not paid, whether they are immediately
withdrawn if candidate found, whether the site also helps in recruitment process or just provide space
for ads etc.). The main merit of BGT data is its granularity level. While JOLTS ask a nationally representa-
tive sample of employers about vacancies they wish to fill in the near term, it is typically available only
at aggregated levels, and contains relatively little information about the characteristics of vacancies. In
contrast, the BGT data contain 70 possible standardized fields for each vacancy. Authors use informa-
tion on occupation, geography, skill requirements, and firm identifiers. The codified skills include stated
education and experience requirements, as well as specific skills standardized from the text in each job
posting.

Hershbein and Kahn (2018) provide a description of industry and occupation distributions of vacancies
in BGT relative to other sources (JOLTS, the Current Population Survey, and Occupational Employment
Statistics), an analysis of how these distributions have changed over the sample period, and correlations
between the datasets. Authors state that BGT postings are disproportionately concentrated in occupa-
tions and industries that require higher skill, the distributions are relatively stable across time, and the
aggregate and industry trends in the quantity of job offers track other sources reasonably closely.
In
comparison to JOLTS, this data overrepresents health care and social assistance, finance, insurance, and
education.
It underrepresents accommodation and food services, public administration/government,
and construction. Authors show that education requirements in BGT data strongly correlate with aver-
age education levels of employed workers at the MSA and occupation levels.

Carnevale et al. (2014) show that the occupation-industry composition of the BGT data are similar to that
of the Conference Board’s HWOL. Moreover, the authors audited a sample of job postings in the BGT
database and compared them to the actual text of the postings, finding that the coding for occupation,
education, experience were at least 80% accurate.

Rothwell (2014) compares the occupational distributions of BGT data to those from state vacancy sur-
veys for selected metropolitan areas for which data are available. He finds that computer, management,
and business occupations are overrepresented in comparison to the state vacancy surveys, while health
care support, transportation, maintenance, sales, and food service workers are underrepresented. Fur-
thermore, it is said that BGT regularly revises and attempts to improve its algorithms (applying them
retroactively on the complete historical database of postings).

Turrell et al. (2018) and Turrell et al. (2019) transform the text of job adverts into time series data la-
belled by official classifications (Standard Occupational Classification codes). They deal with flow data
and transform these data into stocks. Their data consist of millions of individual online job ads posted
by firms and recruitment agencies on the website Reed.co.uk in the UK. The site facilitates matching be-
tween firms and jobseekers. In contrast to webcrawling algorithms, on this website companies post their
vacancies directly with recruiters. Recruiters may have access to private information about the job va-
cancy which an aggregator would not, e.g. the salary offered. This is an advantage of using only one
website to analyse online job offers trends, with an obvious drawback of lower amount of job offers. Au-
thors process, clean, re-weight, and use this data as a measure of job vacancies by occupation and by
region over time, and according to existing official statistical classifications.

They report many sources of bias in online job offers as vacancy measures. They adjust “aggregate out-
flow bias” with data on an average duration of job offer in the UK economy (2-4 weeks) instead of 6 weeks
as it is on the website they use. They report a “differential outflow bias” as the one connected to varying
duration of vacancies according to occupation. In contrary, in the OJA dataset job offer duration is not
fixed and potentially unknown because Cedefop uses many websites. Turrell et al. (2019) treat coverage
bias by applying weights. They reweight data by occupation and sector of activity over time. They also
state that “the ads are run at a cost for the posting party so the concerns about an ever-growing stock of
vacancies which have, in reality, been filled or withdrawn do not apply.”

Inferring job vacancies from online job advertisements

12

General quality issues

2

Turrell et al. (2019) point that some biases in survey-based research are non-existent in online job offers.
These are non-response bias, incomplete-response bias, and overestimation of the vacancies posted by
large companies. Probability-based surveys also possess the “frequency mismatch”, because vacancy
statistics in the EU is based on a quarterly survey, while unemployment statistics, at least to some extent,
is available on a monthly basis. Small vs. large firms’ bias may take a different form in online job offers
than in probability-based surveys. It seems reasonable to assume that both types of companies publish
job ads online. However, large firms may use paid websites more often.

The newly created vacancy time series, split by occupation, are compared by Turrell et al. (2019) to exist-
ing data on UK job vacancies, namely the ONS’ Vacancy Survey and JobCentre Plus data. To demonstrate
the utility of their analysis, authors use data they obtain to estimate Beveridge curves by occupation and
to calculate the rate of mismatch unemployment (by occupation) for the UK.

Inferring job vacancies from online job advertisements

13

Methods

3

3 Methods

3.1 Basic settings

The main goal of the project described in this publication was to develop estimators for the number of
job vacancies from data on online job advertisements, taking into account the differences in definition
of statistical units, coverage and selection bias.

Therefore, from the formal perspective the goal is to estimate total number of job vacancies given by

NX

Nt =

yit,

i=1

(1)

where yit = 0, 1, 2, ... is number of job vacancies in the company i at the end of given quarter t. This
is the standard way where instead of sampling target population we sample a reference population by
means of indirect sampling. See Deville and Lavallée (2006) andLavallée (2009) for more details.

Currently, National Statistical Institutes provide estimates of quantity (1) on the basis of sample surveys
and using Horwitz-Thompson estimator

nX

Nt =

dityit,

i=1

(2)

where dit is the design weight denoting inverse of probabilities of inclusion π−1
it . However, these surveys
suffer from non-response and coverage errors therefore, dit is adjusted by means of post-stratification
or calibration. After this procedure, dit is replaced by wit and equation (2) becomes

nX

Nt =

wityit.

i=1

(3)

An alternative way of estimating the total number of job advertisements might be to use the naïve esti-
mator given by

Nt = p−1

t N Internet
t

,

(4)

where pt is the share of job vacancies published online (this may be obtained from sample surveys) and
N Internet
is the total number of vacancies published online. However, obtaining pt requires additional
costs and N Internet
should be obtained without errors. Moreover, all the cases above require access to

t

t

Inferring job vacancies from online job advertisements

14

Methods

3

sampling frames/lists that cover the target population. This assumption is rather naïve, as we do not
have these sources and thus in this chapter we discuss alternative ways of estimating population size
based on non-probability data.

3.2 Unit-level methods

3.2.1 Modelling misclassification

Following Liu and Zhang (2017), let Y to represent the true state of the binary response variable which
could be modelled by the generalized linear model (i.e. logistic regression) and X is matrix of predictors.
Then let {(˜yi, xi)} be a set of data collected from n participants where ˜yi are realisations of the poten-
tially misclassified variable ˜Y . Under the assumption of non-differential misclassification, the chance of
misclassification is only related to the true status yi by the transition probability distribution function

Pr (˜yi = 1|yi = 0) = r0,
Pr (˜yi = 0|yi = 0) = 1 − r0,
Pr (˜yi = 0|yi = 1) = r1,
Pr (˜yi = 1|yi = 1) = 1 − r1,

(5)

where r0 and ri are called false positive (FP) and false negative (FN) rates, respectively, representing the
extent of misclassification.

It follows that the regular logistic regression model can be extended to include both false-positive and
false-negative misclassification parameters

8
>>><
>>>:

yi ∼ Bern (πi) ,
πi = r0 + (1 − r0 − r1) Fi,
1
Fi =
1+exp(−ηi) ,
ηi = β0 + β1x1i + · · · + βpxpi.

(6)

To estimate the parameters of the logistic models, the maximum likelihood (ML) estimation method
is used because it provides consistent and asymptotically unbiased parameter estimates and standard
error estimates.

The algorithm is based on the estimated equations from the ML estimation. The probability density
function of ˜Y , conditional on the covariates xi is

(cid:16)

(cid:17)

Pr

˜Yi = ˜yi|xi

= πyi

i (1 − πi)1−yi = exp {yiθi − log (1 + exp (θi))} ,

(7)

where θi = log πi
1−πi
observations the likelihood function is

, πi = r0 + (1 − r0 − r1)Fi, Fi = exp(ηi)

1+exp(ηi) and ηi = x′

iβ. Given n independent

L = exp

(

nX

nX

yiθi −

i=1

i=1

)

log (1 + exp (θi))

,

and the corresponding log-likelihood

Inferring job vacancies from online job advertisements

(8)

15

Methods

3

nX

nX

l =

li =

i=1

i=1

[yiθi − log (1 + exp (θi))] .

(9)

This model requires information about which job advertisements are erroneous, outdated, out-of-scope
or not job vacancies and auxiliary variables X are required.

In the literature further developments of these methods could be found. For instance, one can refer to
Roy et al. (2005), Daniel et al. (2005), Meyer and Mittag (2017), Pires and Quinino (2019) and Roy et al.
(2013) to name few.

3.2.2 Non-ignorable selection

Literature on non-ignorable non-response and selection bias is rich (see e.g. Pfeffermann and Sikov
(2011), Riddles et al. (2016), Chang and Kott (2008) and Kott and Chang (2010)). Sikov (2018) and Tang
and Ju (2018) provide a recent review of some approaches. In general, distinction between these meth-
ods is based on the delimitation of two cases:

1.

information about characteristics of non-respondents (not covered units) available,

2. only information about characteristics of selected units is available,

and for both cases it is assumed that known population totals are available.

In this paper we will focus only on the second case as only online data from Cedefop is available.
particular, we will focus on the model proposed by Chang and Kott (2008) and Kott and Chang (2010).

In

Let ci = (c1i, ..., cKi) denote the values of the calibration variables for unit i that was in the non-
probability sample. To estimate the unknown parameters by forming the non-linear regression equa-
tions

CP OP =

rX

i=1

wi

ci
ρ(yi, vi; γ)

+ ϵ∗,

(10)

P

N

j=1 cj, ϵ∗ is a vector of errors, ρ(yi, vi; γ) = P (Ri = 1|yi, vi; γ), and wi = π−1
where CP OP =
denote the sampling weights. The wi are assumed to be known for every responding unit in the sample.
Chang and Kott (2008) proposed an iterative procedure to obtain γ

i

ˆγk+1 = ˆγk + { ˆH(ˆγk)T V −1(ˆγk) ˆH(ˆγk)}−1 ×

CP OP −

rX

i=1

wi

ci
ρ(yi, vi; ˆγk)

!

,

where

ˆH(ˆγk) =

∂wi

ci
ρ(yi,vi;γ)
∂γ

|γ = ˆγk

(11)

(12)

and V −1(ˆγk) is the inverse of an estimator for the quasi-randomisation variance wi
puted at γ = ˆγk.

ci

ρ(yi,vi;γ) of com-

Having estimated the response probabilities, the use of this approach allows estimating the population
totals of the target variables of interest, but it does not allow imputation of the missing data, because no
model is assumed for the outcome values.

Inferring job vacancies from online job advertisements

16

 
Methods

3

This method may be applied to reduce bias due to duplication of records. For instance, the yi may be
the number of advertisements and xi may be the characteristics of companies with known population
totals CP OP of these with free vacancies.

3.2.3 Single source capture-recapture

In case when only one data source is available a common approach is to analyse repeated observations
of given persons or objects by applying regression models. In such case, when the goal is to estimate
population size, we apply single source capture-recapture approach.

Let us assume that dataset A contains units i that are members of the target population U . In this ap-
proach, we assume that dataset A is error free i.e. no erroneous units or over-coverage are present, but
it allows to identify multiple recordings of the same units. Thus, the dataset contains multiple actions
for the same unit.

Example of data is presented in table 1 where column Captures refers to number of occurrences, i.e. 1 –
only one registration, 2 – two registrations, and so on. Column Number of units refers to the number of
units that were observed once, twice, or more times. Note that this table does not contain information
about zero captures N0 and should be estimated (i.e. N = N0 + Nobs).

Table 1: Example of single source capture-recapture data

Captures Number of units (N)

0
1
2
3
4
6
Total

– (N0)
1500
40
5
2
1
N = N0 + 1548

Source: own elaboration.

To tackle this problem, i.e. estimate missing number of units, a flexible approach based on distributional
assumptions about number of captures is applied. In particular, zero-truncated count regression models
are applied, for instance zero-truncated Poisson or zero-truncated negative binomial regression. These
models often use covariates such as sex, age, or nationality to account for heterogeneity in captures. For
recent review of approaches see Bohning et al. (2017) and Zhang and Chambers (2019).

Let us focus on the baseline model, where the number of apprehensions/captures of a member of the
target population, denoted by yi for i = 1, ..., N , follows a Poisson distribution with parameter λ. We
have

P (yi = 0) = e−λ and P (yi|yi > 0; λ) =

P (yi; λ)
P (yi > 0; λ)

=

e−λλyi
yi! (1 − e−λ)

.

(13)

The parameter λ can be estimated based on the observed N1, N2, ... denoted by ˆλ under the truncated
Poisson distribution. An estimated Horvitz-Thompson estimator is given by

ˆNHT =

Nobs
1 − e−ˆλ

=

NobsX

i=1

P

(cid:16)

1
yi > 0; ˆλ

(cid:17) .

However, this approach has some drawbacks that were discussed by Zhang (2008):

Inferring job vacancies from online job advertisements

(14)

17

Methods

3

• captures are not independent – positive contagion occur if previous apprehensions increase the
probability of subsequent apprehensions; whereas negative contagion occur if the probability de-
creases.

• unexplained heterogeneity – the homogeneity assumption is violated if there are differences in

the individual Poisson parameters that cannot be explained by the observed covariates.

• out-of-sample units are like in-sample units – other people with the same x must be outside the

sample.

• closeness of the population – population is clearly not closed during the data collection period.
The concepts of exposure and hit rate, people with different life duration in the population should
have different Poisson parameters.

Recently, to overcome some of these limitations, Godwin and Böhning (2017) proposed zero-truncated
(positive) one-inflated Poisson regression and Böhning and van der Heijden (2019) showed the identity
of the zero-truncated, one-inflated likelihood and the zero-one-truncated likelihood for general count
densities. Godwin and Böhning (2017) justified, one-inflation in two ways:

1. The subject discovers that the observational experience is more unpleasant than expected and

decides to put forth avoidance effort towards being observed again.

2. Additionally, the subject may learn how to avoid being observed subsequently, which we call

avoidance ability.

Let us consider two distributions. Let fy(λ) be the mass function for the Poisson distribution. Then, a
one-inflated Poisson distribution is

8
><

>:

fy(λ),
ω(1 − f0(λ)) + (1 − ω)fy(λ),
(1 − ω)fy(λ),

y = 0,
y = 1
y > 1,

(15)

where f0(λ) is the probability that a 0 occurs under a Poisson distribution and ω is the proportion of
zero-inflation.

Zero-truncating expression (15) results in the following OIPP distribution:

f OIP P
1

= ω + (1 − ω)

λ
exp(λ) − 1

y = 1

f OIP P
y

= (1 − ω)

λy
(exp(λ) − 1)y!

y = 2, 3, ...

and the resulting estimator of population size is given by

ˆN OIP P =

Nobs
1 − exp ˆλ

,

(16)

(17)

where Nobs is the number of observed units.

Note that this approach was mainly applied for the hard-to-reach populations and suitability for infer-
ring the total number of job vacancies should be verified. However, one should note that this approach
requires error-free data i.e. no erroneous records or out-of-scope units.

Inferring job vacancies from online job advertisements

18

Methods

3

3.2.4 Multiple source capture-recapture

3.2.4.1 CAPTURE-RECAPTURE FOR DEPENDENT SOURCES

Kiranmoy Chatterjee and his colleagues in the series of papers on dual and triple estimation (Chatterjee
and Bhuyan (2017), Chatterjee and Mukherjee (2018), Chatterjee and Bhuyan (2019a) and Chatterjee and
Bhuyan (2019b)) proposed estimators that take into account dependence between sources.

Let us focus on Chatterjee and Bhuyan (2017) with two lists. Let (Y, Z) be a paired variable such that Yi
and Zi denote the List 1 and List 2 inclusion status of the ith individual belonging to U . This is presented
in the Table 2. Note that standard Dual-System Estimation have the following assumptions

(S1) Population is closed until the second sample is taken.

(S2) Individuals are homogeneous with respect to their capture probabilities.

(S3) Inclusion of each individual, belonging to U , in List 2 is causally independent to its inclusion in List

1.

Table 2: Dual-record-System (DRS)

List 1

In
Out
Total

In
x11[p11]
x01[p01]
x.1[p.1]

List 2

Out
x10[p10]
x00[p00]
x.0[p.0]

Total
x1.[p1.]
x0.[p0.]
x.. = N [p..]

Source: Chatterjee and Bhuyan (2017, p. 5).

Having that in mind, Chatterjee and Bhuyan (2017) introduced Bivariate Bernoulli model (BBM). Suppose
that (Y, Z) follows bivariate random variables distributed as

(

(Yi, Zi) ∼

(X1, X2) with prob. 1 − α,
(X1, X2) with prob. α,

(18)

where X1 and X2 are independently distributed Bernoulli random variables with parameters p1 and p2,
respectively. Let pyz = P r(Y = y, Z = z) for y, z = {0, 1}. Thus, based on the parameters involved in
the above model (1), we have the following cell probabilities in the DRS

p11 = αp1 + (1 − α)p1p2,
p10 = (1 − α)p1(1 − p2),
p01 = (1 − α)(1 − p2)p2,
p00 = α(1 − p1) + (1 − α)(1 − p1)(1 − p2).

Consequently, the marginal probabilities are

pY = p1. = p1,
pZ = p.1 = αp1 + (1 − α)p2,

(19)

(20)

with Cov(Y, Z) = αp1(1 − p1). This introduces Bivariate Bernoulli model for Dual-record System (BBM-
DRS).

Inferring job vacancies from online job advertisements

19

Methods

3

Based on that, Chatterjee and Bhuyan (2017) proposed two models. We will focus only on model 2. Let
us assume that U of size N can be divided into two mutually exclusive and exhaustive sub-populations
UA and UB with size NA and NB (i.e. stratified populations).

In the model 2, Chatterjee and Bhuyan (2017) relaxed the assumption (S3) and considered BBM-DRS for
both UA and UB with parameters p1 = p1k, p2 = p2k, α = αk, N = Nk for k = A, B. Additionally,
αA = αB = α0, because we have two lists, so we have only one value of correlation between these data
sources.

Chatterjee and Bhuyan (2017) considered two methods to estimate vector of parameters (NA, NB, p1, p2A, p2B, α0)
in this model. Method of moments solution is given by (21)

x01B (x1·Ax10B − x1:Bx10A)
x1·B (x01Ax10B − x10Ax01B)
x01A (x1·Ax10B − x10Ax10A)
x1·A (x1Ax10B − x10Ax10A)
1
1 − ˆp2A
x1·A
(cid:16)
1
ˆp2A

− 1

(cid:17)

ˆp2A =

ˆp2B =

ˆα0 = 1 − x10A
x1·A

ˆp1 =

ˆNA =

ˆNB =

1 + x1·A
x10A
x1·B
ˆp1
x1.B
ˆp1

and Maximum Likelihood solution is solution of the following likelihood function (22)

L (θ|xA, xB) ∝

[α0p1 + (1 − α0) p2A]x11A

NA!NB!
(NA − x0A)! (NB − x0B)!
× [α0p1 + (1 − α0) p1p2B]x11B p(x10A+x10B )
× px01A
× [α0 (1 − p1) + (1 − α0) (1 − p1) (1 − p2A)](NA−x0A)
× [α0 (1 − p1) + (1 − α0) (1 − p1) (1 − p2B)](NB −x0B ) ,

2A px01B

1

(1 − p1)(x01A+x01B )

2B (1 − p2A)x10A (1 − p2B)x10B (1 − α0)(x10A+x01A+x10B +x01B )

(21)

(22)

where xk = (x11k, x10k, x01k) and x0k = x11k + x10k + x01k, for k = A, B.

In a recent paper, Chatterjee and Bhuyan (2019a) extended the Bivariate Bernoulli model to a dependent
triple-record system. Note that, both approaches assume no duplicates and erroneous records (i.e. no
over-coverage).

3.2.4.2 TRIMMED DUAL SYSTEM ESTIMATION

Zhang (2015) considered the case of two data sources where the first suffers from over- and under-
coverage and the second (i.e. independent survey) suffers only from under-coverage. Note that, Zhang
(2015) states An additional independent coverage survey with only undercoverage error is always needed for
estimation. Further, Zhang and Dunne (2017) proposed Trimmed Dual system estimation (TDSE) and we
will focus on this approach.

Let N be the unknown size of the target population, denoted by U . Let A be the first list enumeration
that is of size x. Suppose list A is subject to over-coverage, and the number of erroneous records is r,
i.e., the size of set {i; i ∈ A and i ̸∈ U }. Suppose list A is subject to under-coverage as well, so that

Inferring job vacancies from online job advertisements

20

Methods

3

x − r < N . Let B be the second list enumeration that is of the size n. Suppose list B is subject to only
under-coverage, so that n ≤ N , but there are no erroneous records in B.

Suppose the records in lists A and B can be linked to each other in an error-free manner, which we refer
to simply as the assumption of matching. This is a very common assumption, although it can be difficult
to satisfy in practice if the two lists do not share a unique identifier. However, the linkage errors are not
easy to adjust. For now, suppose that error-free matching between A and B gives rise to the matched
list AB with m records.

Because one does not actually know r, i.e., the number of erroneous records in A. But one can (a) score
some records in list A, which are most likely to be erroneous, (b) match them to list B, and then (c)
calculate the DSE as if list A would have been free of erroneous enumeration once the scored records
had been removed. This yields what we call the trimmed DSE, given by

ˆNk = n

x − k
m − k1

,

(23)

where k is the number of scored (verified) records in list A (of size n, and k1 is the number of records
among them that can be matched to list B (of size x). The naïve DSE estimator of N would be in this case

ˆN = nx/m.

(24)

To summarise, as long as one is able to score the erroneous records in list A more effectively than random
scoring and one does not score more records than the total number of erroneous records in list A, the
trimmed DSE (23) can be expected to reduce the bias of the naïve DSE and move it closer to the ideal
DSE.

Variance of (23) obtained by linearization is given by

ˆV ( ˆNk) = n(n − mk)xk(xk − mk)/m3
k,

(25)

where xk = x − k and mk = m − k1.

Now, consider that source B also suffers from over-coverage. The second estimator considers the case
when both sources contain over-coverage

ˆNk =

(n1 − r1)(n2 − r2)
n12 − r12

,

(26)

where r1, r2 and r12 be the number of erroneous records in list A (of size n1), B (of size n2) and AB (of
size n12), respectively.

Finally, Zhang and Dunne (2017) consider record linkage errors when identifiers are not available. Con-
sider the case when erroneous enumeration is only present in list A. Let mL be the number of records
in the linked list AB. Given the existence of linkage errors, let u be the number of missed matches, and
let e be the number of false links. In other words, the true number of matches between A and B is given
by

m = mL − e + u.

(27)

Let f be the rate of missing (matches) and q is the rate of false links and ˆξ = (1 − ˆq)/(1 − ˆf ) then a
trimmed LDSE can possibly be given by

Inferring job vacancies from online job advertisements

21

ˆNk =

n(x − k)
ˆξ(mL − k1L)

,

Methods

3

(28)

where k is the number of records scored in A (of size n) and k1L that among the linked list AB, and
ˆξ = (1− ˆq)/(1− ˆf ) is based on the estimated linkage error parameters; f is the rate of missing (matches)
and q is the rate of false links. But it is impossible to conclude on the properties of the trimmed LDSE
without some strong additional assumptions involving the linkage errors.

Zhang and Dunne (2017) applied trimming based on subjectively identifying those records that are most
likely to contain erroneous information. In this case, the trimming method removes records for persons
in list A in several steps, when a person has only an employment record with lower remuneration than
a specified amount in EUR. For instance, they consider the following steps:

1. step 1 requires removing records for persons with only an employment record with earnings lower

than 1000 EUR,

2. step 2 removes records for persons with only an employment record with earnings lower than 2000

EUR,

3. and so on.

Note that Zhang and Dunne (2017) mainly focus on situation when over-coverage is a result of errors or
out-of-scope units but not duplicated records.

Further advances and readings may include:

• Capture-recapture methods in the presence of linkage errors – Zhang and Chambers (2019, Chap-

ter 3),

• Estimating population size in multiple record system with uncertainty of state identification –

Zhang and Chambers (2019, Chapter 8),

• Log-linear models for erroneous list data – Zhang and Chambers (2019, Chapter 9).

3.3 Domain-level methods

3.3.1 Measurement error models

In general, for the measurement error model we assume that

γ = θ + b + ϵ,

(29)

where γ is the proxy measurement (i.e. number of job vacancies based on job advertisements), θ is
the parameter of interest (i.e. true number of job vacancies), b is the bias introduced by the differences
between concepts and finally, ϵ ∼ N (0, σ) is an error.

These models are rather used to estimate the bias and thus require source to evaluate it. For instance,
Lohr and Brick (2012) considered small area (domain) estimation with two data surveys, where one is sub-
ject to measurement error that results with additive or multiplicative bias. Consider the first case, where
¯yd is estimated from the first survey and ¯xd is from the second survey that suffers from measurement
error. Relation between these surveys can be written as in the equation (30)

Inferring job vacancies from online job advertisements

22

Methods

3

(cid:18)

¯yd
¯xd

(cid:19)

(cid:20)(cid:18)

∼ N2

θd
θd + ηd

(cid:19)

(cid:18)

, σ2

1/nyd
0

0
1/nxd

(cid:19)(cid:21)

,

(30)

where θd is the true value, ηd is bias, σ2 is variance and nyd, nxd are effective sample sizes for the first
and the second survey, respectively.

Note that, measurement errors models require existence of the gold standard that measures the concept
without errors. As the JVS in most countries are based on sample surveys, it has measurement error, so
the existing estimates cannot be used to correct bias in the estimates based on online data.

3.3.2 Modelling of under-reporting

3.3.2.1 BAYESIAN ESTIMATION OF UNDER-REPORTED COUNT DATA

Recently, Stoner et al. (2019) proposed a Bayesian hierarchical model for under-counting of Tuberculosis
in Brazil. Their model can be summarised as follows.

• Let yi,t,s be the number of events occurring in the space s ∈ S, time t ∈ T and in domain i (e.g.

age group) .

• If yi,t,s is believed to have been perfectly observed, the counts are conventionally modelled by an

appropriate conditional distribution p(yi,t,s|θ), usually either Poisson or Negative Binomial.

• θ represents random effects allowing for various dependency and grouping structures (e.g., space

and time), as well as parameters associated with relevant covariates.

• In case of under-counting/-reporting instead of yi,t,s we observe zi,t,s and let Ii,t,s be the index

of under-reporting treated as a random variable.

• It is assumed that Ii,t,s ∼ Bernoulli(πi,t,s) but Stoner et al. (2019) assumed that Ii,t,s can be contin-
uous in the range [0, 1] to be interpreted as the proportion of true counts that have been reported.

• Finally, the proposed hierarchical model given by (31)

(cid:18)

log

πi,t,s
1 − πi,t,s

zi,t,s ∼ Binomial(πi,t,s, yi,t,s)
JX

(cid:19)

= β0 +

j=1

βjw(j)
i,t,s

yi,t,s ∼ Poisson(λi,t,s)

log(λi,t,s) = α0 +

KX

k=1

αkx(k)

i,t,s,

(31)

where yi,t,s are unknown true counts that follow Poisson distribution, α and β are unknown vec-
tors of parameters associated with the probability of under-reporting πi,t,s and level of true counts
yi,t,s. Note that we have two sets of independent variables Z associated with πi,t,s and X associ-
ated with yi,t,s. Therefore, this model requires access to powerful variables.

• The final model presented in Stoner et al. (2019) was more complicated than model (31) because
of inclusion of random effects (including spatial autocorrelation), non-linear relationship with Z
and X and offset log(Pi,t,s) was used instead of α0 where Pi,t,s is the population size.

Another approach to correcting under-reporting is to base inference on the censored likelihood. This is
the product of the evaluation of (32)

p(y|z, θ) =

Y

p(yi,t,s|θ)

Y

Ii,t,s=1

Ii,t,s=0

p(yi,t,s ≥ zi,t,s|θ).

Inferring job vacancies from online job advertisements

(32)

23

Methods

3

In this framework, the indicator Ii,t,s for which data are under-reported is binary (where Ii,t,s = 1 when
zi,t,s = yi,t,) and notation is as in Stoner et al. (2019). Strength of this approach is that all of the observed
counts contribute to the inference and, by accounting for the under-reporting in the model design, a
more reliable inference on θ is obtained. However, information on which counts are under-reported is
not always readily available, introducing the challenge of having to determine or estimate this classifi-
cation. Oliveira et al. (2017) presented an alternative to this approach, which treats the binary under-
reporting indicator Ii,t,s as unobserved and therefore random.

Based on this approach the following requirements about the data should be underlined

• zi,t,s should be free of duplicated records,

• strong set of correlates Z and X are required,

• information about the reference population P may be needed.

Fortunately, the model is easily implemented in R (R Core Team, 2019) using stan (Carpenter et al., 2017)
or brms (Bürkner, 2017).

3.3.2.2 ESTIMATION OF HIDDEN POPULATION BASED ON DOMAIN DATA

Now, we focus on an alternative approach proposed by Zhang (2008) based solely on an aggregated
data assuming relationship between the size of registered (observed) and unregistered (unobserved)
population. This relationship is further modelled by generalized non-linear regression model, in partic-
ular Gamma-Poisson hierarchical model. In the paper, Zhang (2008) focused on estimation of irregular
migration but we adapt the description and notation to online data.

For both the target and the reference populations, let i = 1, ..., t be the index of the sub-population
classified by the country of citizenship and origin, respectively. Assume that the observed number of
irregular residents follows a Poisson distribution, with parameter λi, denoted by

mi ∼ Poisson (λi) .

(33)

It is intuitively plausible that the parameter λi should depend on two other quantities: (a) the total num-
ber of irregular residents from country i, denoted by Mi, and (b) the probability of being observed, i.e.
the probability for an irregular resident denoted by pi, i.e. λi = Mipi. In addition, let ui = Mipi/E(Mipi|ni, Ni)
where E(Mipi|ni, Mi) denotes the conditional expectation of Mipi given ni and Ni. The ui is a random
effect that accounts for heterogeneous variation from one country to another. Together, we have

λi = µiui, where µi = E (Mipi|ni, Ni) = E (Mi|Ni) · E (pi|Mi, ni, Ni) .

(34)

We complete the model specification by assuming that

ξi = E (Mi|Ni) = N α
i ,

E (pi|Mi, ni, Ni) = E (pi|ni, Ni) =

(cid:18)

(cid:19)β

,

ni
Ni

ui ∼ Gamma(1, ϕ),

The target parameter and its estimator are given as, respectively,

Inferring job vacancies from online job advertisements

(35)

(36)

(37)

24

and

ξ =

tX

i=1

E (Mi|Ni) =

X

i

N α
i ,

ˆξ =

X

i

N ˆα
i ,

Methods

3

(38)

(39)

where ˆα is the estimator of α. We shall use the maximum likelihood estimator (MLE). Denote by L(η, m)
the likelihood of η = (α, β, ϕ) given mi, for i = 1, ..., t. Under the Poisson gamma model, we have

f (mi, ui; η) =

e−µiui (µiui)mi
mi!

· ϕϕuϕ−1
i
Γ(ϕ)

e−ϕui

=

µmi
i ϕϕ
mi!Γ(ϕ)

e−ui(µi+ϕ)umi+ϕ−1

i

,

(40)

where Γ() is the gamma function. Thus,

Z ∞

f (mi; η) =

f (mi, ui; η) d (ui)

0
µmi
i ϕϕ
mi!Γ(ϕ)
µmi
i ϕϕ
mi!Γ(ϕ)

=

=

Z ∞

0

√

e−(

ui)2(µi+ϕ) (

√

ui)2(mi+ϕ−1) 2

√

√

uid (

ui)

(41)

(µi + ϕ)

−(mi+ϕ) Γ (mi + ϕ) ,

ui and k = 2(mi + ϕ) − 1. Notice
based on the identity
that, conditional on mi, ui has the gamma distribution with mean (mi + ϕ)/(µi + ϕ) and variance
(mi + ϕ)/(µi + ϕ)2.

zkdz = 1

, with z =

k+1
2

2 Γ

2 γ− k+1

R ∞
0 e−γz2

(cid:0)

(cid:1)

√

The likelihood is given by

L(η; m) =

tY

i=1

f (mi; η) ,

The log-likelihood is thus, disregarding constant terms, given by

tX

l(η; m) =

li(η),

i=1

where

li(η) = mi log µi − (mi + ϕ) log (µi + ϕ) + log Γ (mi + ϕ) + ϕ log ϕ − log Γ(ϕ)

.
= mi log µi − (mi + ϕ) log (µi + ϕ) + ϕ log ϕ
+ (mi + ϕ − 0.5) log (mi + ϕ) − (mi + ϕ) − (ϕ − 0.5) log(ϕ) + ϕ
= mi log µi − (mi + ϕ) log (µi + ϕ) + (mi + ϕ − 0.5) log (mi + ϕ) + 0.5 log ϕ,

by the Stirling approximation log Γ(z)

.
= (z − 0.5) log(z) + 0.5 log(2π) − z.

Inferring job vacancies from online job advertisements

(42)

(43)

(44)

25

Methods

3

The mean parameter µi is linear on the log scale, denoted by logµi = xT
ates xi and parameters γ. Now that li(η) depends on γ only through µi, we have

i γ with generic vector of covari-

∂li(η)
∂γ

=

∂li(η)
∂µi

∂µi
∂ log µi

∂ log µi
∂γ

=

∂li(η)
∂µi

µixi =

mi − µi
µi + ϕ

ϕxi,

where ∂li(η)/∂µi = mi/µi − (mi + ϕ) / (µi + ϕ), and

∂li(η)
∂ϕ

= − log (µi + ϕ) − mi + ϕ
µi + ϕ

+ log (mi + ϕ) +

mi + ϕ − 0.5
mi + ϕ

+

1
2ϕ

.

Moreover,

and

∂2li(η)
∂γ∂γT =

∂2li(η)
∂µ2
i

µixi

∂µi
∂γT +

∂li(η)
∂µi

xi

∂µi
∂γT = −

(cid:18)

(cid:19)

xixT
i ,

mi + ϕ
µi + ϕ

ϕ

∂2li(η)

∂ϕ2 = − 2µi + ϕ − mi
(µi + ϕ)2 +

mi + ϕ + 0.5
(mi + ϕ)2

− 1

2ϕ2 ,

∂2li(η)
∂γ∂ϕ

(cid:18)

(cid:18)

=

∂

∂li(η)
∂µi

(cid:19)

(cid:19)

/∂ϕ

µixi = − µi − mi

(µi + ϕ)2 µixi =

(cid:18)

(cid:19)T

.

∂2li(η)
∂ϕ∂γT

The MLE of η, denoted by ˆη, is given by the solution to the likelihood equations, i.e.

∂l(η; m)
∂η

=

tX

i=1

∂li(η)
∂η

= 0.

(45)

(46)

(47)

(48)

(49)

(50)

The MLE can be obtained using the Newton-Raphson method. As the starting values we use the ordinary
least squares fit of the heuristic log-ratio model. We use the estimated α and β as the starting values for
the same parameters of the Poisson-gamma model, and the inverse of the estimated V (ϵi) as the starting
value for ϕ. We estimate the model using maxLik package in R software.

3.4 Summary

This chapter summarised the literature on population size estimation in the presence of non-sampling
data, in particular under- and over-coverage. Most of these methods require information about auxiliary
variables from the study population, audit sample to assess quality or independent data source that is
source of error-free records or dependent variables. Moreover, in case of non-probability data estimation
and assessing uncertainty are based on the model-based assumptions and therefore require simulation
studies.

Inferring job vacancies from online job advertisements

26

Comparing OJA data with job vacancy statistics

4

4 Comparing OJA data with job

vacancy statistics

4.1 Aligning job advertisements data to Job Va-

cancies Statistics

The final goal of the study was to assess methods to estimate the total number of job vacancies based on
online job advertisements (OJA) data. In order to render OJA data comparable with currently published
official statistics we tried to approximate OJA data to the definition of job vacancy used in job vacancies
statistics, as well as to align the type of variable (from flow to stock) and the reference period. We followed
the OJA definitions used by Cedefop as well as their deduplication process.

The relation between job advertisements, or job offers (we will use both terms as having the same mean-
ing), and job vacancies is comparable to the one between family and household. They look similar but
the definition is in fact different. Actually, the definition taken from the Cambridge dictionary(6) states
that:

Cambridge’s job advertisement definition

Job advertisement is an announcement in a newspaper, on the Internet, etc. about a job that people can
apply for.

Cedefop uses a similar definition of job advertisement.

Cedefop’s job advertisement definitions

• The advertisement, defined as the job offer published by a company to search for a new em-

ployee.

• The word ’job vacancy’ in this document means the document (typically in HTML) that describes

the job offer.

Having that in mind, one should align the definitions of job offer and of job vacancy. Cedefop describes
the process as follows:

1. The generation process of vacancies from advertisements is called expansion.

(6) Source: https://dictionary.cambridge.org/pl/dictionary/english/job-advertisement

Inferring job vacancies from online job advertisements

27

Comparing OJA data with job vacancy statistics

4

2. Expansion is not necessary for all analysis dimensions but only for those considered distinguishing
for the job offer (e.g. expansion can be performed for the profession but not for the contract).

3. There are two possible ways of expanding advertisements:

• By selection: choosing one of the possible jobs offers found in the advertisement
• By repetition: generating as many vacancies as the number of jobs offers found in an adver-

tisement.

4. Cedefop also considers using expiry date to align with job vacancy definition: ”As discussed in
Section 6 a critical parameter that has to be considered in assessing the data quality is the identi-
fication of the expiry date of the posted vacancy. This information allows us to define the average
duration of vacancies which is necessary to construct a flow measure comparable to the standard
definition adopted by statistical offices. In the data used in this preliminary experiment the aver-
age duration of a vacancy is approximately one month (39.25 days).”

Job advertisements (including those online) and job vacancies from a probability-based survey are two
different measures of job openings. They cover different populations. Job vacancies and job advertise-
ments might also differ by the definition of a job opening. A job advertisement may not refer to a job
from the labour law point of view, and we are not certain whether a company is convinced that they
want to hire a worker. But after removal of non-work job advertisements and assuming that a company
publishes a valid job advertisement (e.g. credible websites with job advertisements are used), online job
advertisements by definition are just one way to present a job vacancy.

In order to align the type of variable (from flow to stock) and the reference period of the OJA data to
make it comparable to job vacancies statistics, we identified those advertisements that were active at
the end of each quarter:

• 2018Q3 = expire_date >= ”2018-09-30” & grab_date <= ”2018-09-30”,

• 2018Q4 = expire_date >= ”2018-12-31” & grab_date <= ”2018-12-31”,

• 2019Q1 = expire_date >= ”2019-03-31” & grab_date <= ”2019-03-31”,

• 2019Q2 = expire_date >= ”2019-06-30” & grab_date <= ”2019-06-30”,

• 2019Q3 = expire_date >= ”2019-09-30” & grab_date <= ”2019-09-30”,

• 2019Q4 = expire_date >= ”2019-12-31” & grab_date <= ”2019-12-31”,

We also calculated the number of days between the grab_date and the end of quarters:

• 2018Q3_days = ”2018-09-30” - grab_date,

• 2018Q4_days = ”2018-12-31” - grab_date,

• 2019Q1_days = ”2019-03-31” - grab_date,

• 2019Q2_days = ”2019-06-30” - grab_date,

• 2019Q3_days = ”2019-09-30” - grab_date,

• 2019Q4_days = ”2019-12-31” - grab_date.

After this transformation, we got the following results.

• number of advertisements not active at the end of any quarter – 8,183,067.

Inferring job vacancies from online job advertisements

28

Comparing OJA data with job vacancy statistics

4

• number of advertisements active at the end of one single quarter – 44,799,353,

• number of advertisements active at the end of two quarters – 14,697,645,

Table 3 presents the number of advertisements in the OJA dataset at the end of each quarter. A signif-
icant decrease is observed between 2019Q1 and 2019Q2. For instance, in Germany the number of job
offers decreased by 2 million during this period. This type of decrease is not visible in job vacancy statis-
tics compiled by National Statistical Institutes (NSIs) in Member-States. As the data for the same period
for 2018 is not available, we are not sure what is the reason for this decrease. Such unexpected changes
will have significant impact on estimates of job vacancies.

Table 3: Number of advertisements active at the end of each quarter

Country
Austria
Belgium
Bulgaria
Croatia
Cyprus
Czechia
Germany
Denmark
Estonia
Greece
Spain
Finland
France
Hungary
Ireland
Italy
Lithuania
Luxembourg
Latvia
Malta
Netherlands
Poland
Portugal
Romania
Sweden
Slovenia
Slovakia

2018Q3
281,546
305,807
11,919
2,086
1,434
93,145
2,356,289
10,337
1,049
365
226,842
13,975
1,999,566
18,474
105,512
423,393
1,476
10,992
708
2,648
574,150
309,962
40,806
17,074
104,367
2,833
22,710

2018Q4
298,612
367,834
34,977
7,158
2,946
138,361
3,286,338
26,902
4,972
3,870
314,062
27,810
2,660,478
29,377
135,307
576,520
9,832
17,810
6,446
6,222
692,293
320,509
71,498
43,203
196,539
1,499
22,605

2019Q1
364,221
599,394
83,908
22,813
8,175
264,765
4,607,939
87,021
11,314
14,326
532,433
81,143
3,293,666
83,125
180,270
817,059
24,365
23,070
25,751
2,981
864,496
400,852
79,903
134,329
355,811
12,555
63,673

2019Q2
214,306
515,096
60,684
23,398
6,404
239,250
2,618,403
58,743
10,430
13,831
511,229
55,687
2,075,295
60,492
161,850
501,388
24,817
23,355
20,126
3,218
462,977
265,979
68,078
80,192
240,500
15,262
49,356

2019Q3
223,323
452,800
49,132
18,406
7,263
225,002
2,771,006
61,989
8,879
10,380
521,692
60,678
2,821,807
73,386
149,908
730,925
24,912
21,214
16,144
2,689
599,563
402,984
74,404
91,859
158,650
14,709
53,989

2019Q4
138,868
333,538
33,602
19,502
4,298
138,011
1,767,932
59,240
7,707
8,957
411,553
42,939
3,123,865
44,816
111,068
797,023
20,973
18,625
14,929
1,875
752,627
376,891
45,940
67,940
126,353
11,000
44,726

Inferring job vacancies from online job advertisements

29

Comparing OJA data with job vacancy statistics

4

Table 4 presents the number of unique sources of advertisements (websites) in the OJA dataset. We
see that from quarter-to-quarter in 2018 the number of sources increases but after 2019Q1 we see a
decrease. This may be the reason for the decrease of advertisements in 2019Q2.

Furthermore, table 4 further investigates how many data sources were present over the whole period.
The share of unique sources varies between countries. The lowest number of sources observed in all 6
quarters is in Latvia (10.8%), while the highest is in France (48.8%). This indicates that stability of sources
varies and may have further influence on estimates of job vacancies based on job advertisements.

Table 4: Number of unique sources of advertisements in Cedefop’s OJA dataset, by quarter

Country

Austria
Belgium
Bulgaria
Cyprus
Czechia
Germany
Denmark
Estonia
Greece
Spain
Finland
France
Croatia
Hungary
Ireland
Italy
Lithuania
Luxembourg
Latvia
Malta
Netherlands
Poland
Portugal
Romania
Sweden
Slovenia
Slovakia

2018Q3
91
132
111
40
63
147
31
32
17
115
45
152
74
99
119
133
34
53
16
50
134
60
112
117
87
42
76

2018Q4
112
159
137
76
83
171
54
61
35
149
68
186
94
120
132
174
63
68
32
59
155
87
135
139
98
51
99

2019Q1
135
170
167
94
106
186
59
91
49
170
86
205
104
132
158
183
83
63
57
61
187
103
144
176
124
63
117

Number of sources
2019Q3
107
142
126
62
71
156
36
50
39
138
65
192
83
101
126
165
53
62
30
53
155
71
123
141
91
45
83

2019Q4 Whole period
48
84
66
17
30
102
16
15
14
72
20
118
34
49
71
96
21
29
8
20
84
26
63
80
42
20
40

90
138
123
55
52
141
48
44
57
125
61
175
78
91
110
157
49
55
24
45
142
66
113
129
76
45
70

2019Q2
105
160
139
73
87
175
51
65
45
159
66
192
98
120
129
177
67
55
34
50
164
83
132
151
101
49
91

Total
177
210
201
125
144
218
95
116
84
204
124
242
155
174
189
226
117
103
74
102
222
144
183
206
163
95
154

Share
27.12
40.00
32.84
13.60
20.83
46.79
16.84
12.93
16.67
35.29
16.13
48.76
21.94
28.16
37.57
42.48
17.95
28.16
10.81
19.61
37.84
18.06
34.43
38.83
25.77
21.05
25.97

Note: Column Whole period contains information about the number of sources observed in each
quarter from 2018Q3 to 2019Q4. Column Total shows the total number of sources observed in the
period and Share is calculated as values of Whole period divided by Total.

Inferring job vacancies from online job advertisements

30

Comparing OJA data with job vacancy statistics

4

Finally, Table 5 presents co-occurrence of advertisements in several web sources by number of sources.
We report statistics for one, two, three, and four or more data sources. For instance, in Poland 312,218
advertisements were present only in one source, 14,833 in two, 938 in three and 331 in four or more
sources. Thus, we observe one-inflation in the distribution of re-captures of advertisements. We see
that for certain countries such as Croatia, Cyprus, Denmark, Estonia, Greece, Latvia, Lithuania, Malta and
Slovenia the number of re-captures is very small, in most cases lower than 100.

Table 5: Average number of advertisements occurring on one, two, three and four and more
sources advertisements between 2018Q3 and 2019Q4

Country

Number of sources

Austria
Belgium
Bulgaria
Croatia
Cyprus
Czechia
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Ireland
Italy
Latvia
Lithuania
Luxembourg
Malta
Netherlands
Poland
Portugal
Romania
Slovakia
Slovenia
Spain
Sweden

1
229,314
401,652
43,830
15,339
5,009
154,305
50,236
7,324
45,157
2,371,491
2,611,564
7,827
45,909
122,732
544,532
13,961
17,458
14,869
3,184
549,507
312,218
51,508
65,755
39,339
9,562
381,104
189,730

2
9,831
12,952
852
100
36
13,129
228
32
867
114,525
116,131
367
2,617
8,505
38,835
26
119
2,042
42
42,588
14,833
4,792
2,760
1,438
39
16,055
3,200

3
1,213
449
51
6
6
718
6
1
45
15,640
15,134
22
108
280
4,532
2
9
68
2
5,533
938
677
246
118
1
1,618
248

4 and more
200
41
5
2
1
80
1
1
8
3,448
2,795
3
33
20
1,205
0
3
6
0
1,507
331
71
118
94
1
364
58

There is also variability in the number of advertisements observed in two sources in a given quarter.
These sources may vary between quarters and it does not mean that these two websites are the same
over the whole period. What we observe is high variance and varying pattern in number of ads.

One should note that these results depend on the effectiveness of the deduplication procedure applied.

Inferring job vacancies from online job advertisements

31

Comparing OJA data with job vacancy statistics

4

4.2 Relation between online job offers and job

vacancies

4.2.1 Descriptive statistics

Data available at Eurostat was limited to quarterly estimates from job vacancies statistics (JVS). The time
span of our analysis was limited to 2018Q2-2020Q1. However, not all countries supplied data on job
vacancies to Eurostat for the last quarter, which limited our analysis even more. OJA data in this quarter
showed a significant decline (to 20% job ads in 2019Q4 and to 10% of the ones in 2019Q1). One of the
possible reasons for this decline may have been the initial impact of Covid-19 in the labour market. The
OJA dataset also contained few job advertisements for the first quarter. It constituted only 4% of job ads
identified in the consecutive period. This period contained in fact only preliminary results of gathering
job ads. For these reasons, and comparability purposes, we decided to exclude the first and last quarters
from the analysis. Figure 2 compares both datasets for the period finally taken for the analysis, 2018Q3-
2019Q4. It includes aggregated data for the 16 countries for which data from JVS were available. OJA
data are less stable over time. We do observe a lower number of vacancies in 2019Q2 (but not as low as
in the excluded 2018Q2), and then, their significant increase. In 2019Q1 we can see the largest number
of job ads (in contrast to a significant decline in the excluded 2020Q1). Job ads are 2.7 times higher than
job vacancies in 2019Q1. During the rest of the period OJA identifies 40%-91% job offers more than
JVS does. The Pearson linear correlation between the two datasets is 0.61. OJA data show a lot more
variation than JVS with coefficient of variation 0.28 in comparison with 0.03 for JVS and identify more
job offers, with mean 4.6 mln job ads in comparison to 2.5 vacancies per quarter.

Figure 2: Number of job ads in the OJA dataset and number of vacancies from JVS at the end of
quarter for 16 countries

Note: Countries included – Bulgaria, Croatia, Czechia, Estonia, Germany, Hungary, Latvia, Lithuania,
Luxembourg, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Sweden

Large variance is typical for indices that are often used for predicting economic activity, that is leading
indices. Internet data may be more sensitive to economic activity than probability-based surveys. For
this reason seasonal patterns of both statistics may differ as well. Having short time series we cannot
seasonally decompose them, let alone to other unobservable components. This led us to the conclusion
that we cannot analyse time trends (long-term trends and business cycles) of the time series, but only
their general time properties.

Inferring job vacancies from online job advertisements

32

Comparing OJA data with job vacancy statistics

4

Job vacancy statistics were available only for 16 countries, while OJA data contains information on all
countries. In terms of variance OJA data is more comparable to JVS data across countries than over time.
In this case, the coefficient of variation equals 2.47 for OJA data and 2.14 for JVS data. The average num-
ber of job offers across countries differs significantly. Mean OJA number of job offers across all countries
equals 185% of the one from JVS. In the case of the median it is 163%.

For nine countries the number of advertisements identified is larger than the number of vacancies es-
timated by JVS (Figure 3). Large differences between the number of jobs offers in comparison to job
vacancies occurred for Luxembourg, Poland, Netherlands, Bulgaria, and Germany. For these countries
the number of job advertisements were over two times higher than the number of vacancies. Almost
two times lower numbers were estimated for Latvia and Slovenia. The highest similarity was found for
Lithuania (2% dissimilarity).

Figure 3: Mean quarterly number of job ads in OJA data and vacancies in JVS at the end of quarter

Note: Percentages over bars mean by how much OJA data differ from JVS data.

Time trends for each country with both available datasets are presented in Figure 4. Country statis-
tics confirm the general conclusion of aggregate data on large variance of OJA data. Most countries
share similar pattern of job ads with a peak in the middle of the analysed period. Notable exceptions
are Netherlands and Poland. Descriptive statistics of the 96 values for the individual time periods and
countries show that the mean is higher for OJA data by 85% and median is higher by 53%, with more
deviation around them in OJA data (Table 6). Both datasets are leptokurtic, and OJA data is more lep-
tokurtic, meaning that it may contain outliers. Both distributions are similarly positively skewed (right
skewed). They concentrate in lower numbers of vacancies and advertisements.

The definitions of economic activity in OJA data and in JVS differ, even if both are classified using NACE.
While the economic activity in OJA data is based on the vacancy description in the advertisement and
may refer to any of the activities performed by the enterprise, the establishment or even to a smaller
economic unit (e.g. a branch constituted by a separate legal unit), JVS are broken down by the main
economic activity of the enterprise. This may produce differences in results. Despite this, data for NACE
sections are more comparable between OJA data and JVS than data for the total of all economic activities.
However, due to missing data, we were able to include only 13 countries. OJA mean across NACE sections
accounted for 186% of JVS mean, while median accounted for 175% of the one for JVS data. Although
the range was over two times lower for JVS than OJA, the coefficient of variation was comparable and
two times lower than for countries (1.16 for OJA and 0.94 for JVS). The largest differences between the
two datasets were for sections D, M, K, and J, for which OJA data showed three to almost seven times

Inferring job vacancies from online job advertisements

33

Comparing OJA data with job vacancy statistics

4

Figure 4: Number of job ads from OJA data and vacancies from JVS in countries over time

Table 6: Descriptive statistics for OJA data and JVS - countries and time

nbr.val
min
max
range
sum
median
mean
var
std.dev
coef.var
skewness
kurtosis

OJA

JVS

96
708
4607939
4607231
27869238
45378
290305
539026367233
734184
2.53
3.86
15.60

96
6899
1458393
1451494
15092711
29604
157216
107755914226
328262
2.09
3.17
9.00

Note: Countries included – Bulgaria, Croatia, Czechia, Estonia, Germany, Hungary, Latvia, Lithuania,
Luxembourg, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Sweden.

more vacancies (Figure 5). The opposite situation was found for sections F, A, and B, for which OJA data
was lower by 38%-79% than JVS data. Highest similarity was found for sections I, E, R, and G (+/-20%
dissimilarity).

NACE section breakdown (industries) shows a similar time pattern as the country did. JVS data are more
stable, while OJA values rise in the beginning of the analysed period most often reaching a peak in
2019Q1, and then decline (Figure 6). For some sections, OJA mimics changes in JVS, e.g. sections A
and R. Descriptive statistics show us that the median number of job offers in an average NACE section
was 15% lower in OJA data, while the mean number is higher by 86% (Table 7). Again, variance is larger
for OJA data, and even larger than in the country breakdown. The distribution is right-skewed, more for
OJA than JVS, and also more than for the countries. OJA data distribution exhibits a very high kurtosis, a
lot higher than for countries.

For most countries there are no estimates of JVS across occupations nor are uncertainty measures for
recent years being reported. This prevents the comparison between OJA and JVS for ISCO occupations,

Inferring job vacancies from online job advertisements

34

Comparing OJA data with job vacancy statistics

4

Figure 5: Mean quarterly number of job ads from OJA and vacancies from JVS at the end of quarter
across NACE sections

Note: Percentages over bars mean by how much OJA data differs from JVS data.

Figure 6: Number of job ads from OJA and vacancies from JVS in NACE sections over time

Note: Countries included – Bulgaria, Croatia, Czechia, Germany, Hungary, Latvia, Lithuania,
Luxembourg, Netherlands, Poland, Romania, Slovakia, Sweden.

even though OJA data includes detailed data. JVS provides data only for Hungary. At the time of analysis
there are no publicly available data for other countries during 2019.

The occupation identified in OJA has misclassification errors that should be corrected through audit sam-
ples or linking with administrative data (such as jobs at employment offices). Having this in mind, we
provide the comparison of OJA and JVS across ISCO major groups of occupations for Hungary.

Inferring job vacancies from online job advertisements

35

Comparing OJA data with job vacancy statistics

4

Table 7: Descriptive statistics for OJA data and JVS - NACE section and time

nbr.val
min
max
range
sum
median
mean
var
std.dev
coef.var
skewness
kurtosis

OJA

1482
0
991253
991253
27386400
1291
18479
5014878703
70816
3.83
7.95
77.00

JVS

1482
0
273314
273314
14718277
1514
9931
812404965
28503
2.87
5.42
34.08

Note: Countries included – Bulgaria, Croatia, Czechia, Germany, Hungary, Latvia, Lithuania,
Luxembourg, Netherlands, Poland, Romania, Slovakia, Sweden.

For almost all ISCO major groups of occupations OJA data provide lower estimates of job offers in Hun-
gary than JVS do (Figure 7). A notable exception is group 1 (’Managers’), for which OJA data presented
values 3.3-times higher than those in JVS. In the remaining groups the highest discrepancy in the num-
ber of identified jobs offers was for group 6 (’Skilled agricultural, forestry and fishery workers’), where OJA
constituted only 19% of JVS, and group 8 (’Plant and machine operators, and assemblers’), where OJA
constituted 23% of JVS. The lowest differences were identified for group 5 (’Service and sales workers’),
with 3% less ads than vacancies, and group 2 (’Professionals’), with 5% less ads than vacancies. Descrip-
tive statistics show generally larger quantiles of JVS than OJA, but similar coefficient of variation (0.70 for
OJA and 0.67 for JVS).

Figure 7: Mean quarterly number of job ads from OJA and vacancies from JVS for Hungary across
ISCO major groups of occupations at the end of quarter

Note: Percentages over bars mean by how much OJA data differs from JVS.

The comparison of OJA and JVS across occupations in time confirms previous results for countries and
industries on larger coefficient of variation of OJA than JVS (Figure 8). The time pattern was also similar,

Inferring job vacancies from online job advertisements

36

Comparing OJA data with job vacancy statistics

4

with growth in the beginning, peak in the middle of the analysed period and decline in the end, for each
occupational group. In some sections this resembles JVS time trends well (groups 5 and 6), but in other
sections changes of JVS and OJA over time are different.

The distributions of both datasets are quite different (Table 8). The median and mean numbers of job
offers in an average ISCO group are respectively 42% and 36% lower in OJA data than in JVS. The distri-
bution is right-skewed for OJA data, and slightly left-skewed for Eurostat. Kurtosis of OJA distribution is
lower than the one for the normal distribution, and a lot lower than for country and NACE sections. JVS
data distribution is visibly platykurtic.

Figure 8: Number of job ads from OJA and vacancies from JVS for ISCO major groups of occupa-
tions in Hungary over time

Table 8: Descriptive statistics for OJA and JVS data for Hungary - ISCO major groups of occupation
and time

nbr.val
min
max
range
sum
median
mean
var
std.dev
coef.var
skewness
kurtosis

OJA

54
32
21847
21815
309670
5083
5735
24140390
4913
0.86
1.30
1.67

JVS

54
384
19494
19110
485825
8749
8997
33419973
5781
0.64
-0.06
-1.28

4.2.2 Cross-sectional regressions

Figure 9 shows basic features of the relation between OJA and JVS over time. Regressing JVS on OJA gives
a very low, but statistically significant coefficient 0.041 (t-statistic=2.96). Such a low slope parameter
shows incomparability of the fluctuations in the OJA dataset used for this study and that present in JVS.
The residuals are obviously heteroscedastic. To account for this, we need to decompose the dataset into
countries, NACE section (industry), and ISCO occupation. These are the only variables through which we

Inferring job vacancies from online job advertisements

37

Comparing OJA data with job vacancy statistics

4

can currently link OJA and JVS.

Figure 9: Regression fit between JVS and OJA

Note: Countries included – Bulgaria, Croatia, Czechia, Estonia, Germany, Hungary, Latvia, Lithuania,
Luxembourg, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Sweden.

We can observe large differences in OJA-JVS relations between countries. Therefore we calculated sep-
arate regressions for each country (Figure 10). Table 9 shows additional statistics. Significant relations
were found for four countries, namely Czechia, Latvia, Bulgaria, and Sweden. The results for these coun-
tries were satisfactory, with 0.66 < R2 < 0.99. In other countries the slope coefficients were not statis-
tical significant. In six countries the estimated coefficient of slope was negative, even if non-significant.

Figure 10: Regression results between OJA and JVS for individual countries

Inferring job vacancies from online job advertisements

38

Comparing OJA data with job vacancy statistics

4

Table 9: Summary of regression results for individual countries

Intercept

OJA

Bulgaria
Croatia
Czechia
Estonia
Germany
Hungary
Latvia
Lithuania
Luxembourg
Netherlands
Poland
Portugal
Romania
Slovakia
Slovenia
Sweden

17765*** (20.67)
17238** (5.75)
267277*** (325.47)
10954*** (17.56)
1330457*** (11.65)
85817*** (21.07)
21783*** (12.54)
17433*** (15.53)
7338*** (12.82)
288620** ( 8.16)
174627** (5.54)
28243* (4.21)
59478*** (11.33)
26318** (8.16)
18549*** (9.91)
74157** (6.86)

0.047* (2.77)
0.079 (0.46)
0.156*** (36.73)
0.025 (0.33)
0.015 (0.39)
-0.094 (-1.3)
0.354** (3.32)
0.033 (0.58)
0.001 (0.05)
-0.02 (-0.39)
-0.088 (-0.98)
0.074 (0.72)
-0.044 (-0.68)
-0.074 (-1.04)
-0.018 (-0.11)
0.202** (4.00)

R2
0.66
0.05
0.99
0.03
0.04
0.30
0.73
0.08
0.00
0.04
0.19
0.11
0.10
0.21
0.00
0.80

DW test
1.1
1.5
3.3
2.7
2.2
0.9*
1.4
2.7
1.6
0.8
2.2
1.2
1.0
0.9*
1.9
2.6

Similarly to country breakdown, the differences between regression parameters calculated for individual
NACE sections were large. Parameters representing OJA-JVS relation for all but one (O section is the
exception, but the parameter is not statistically significant) are positive (Figure 11). However, only three
out of 19 were statistically significant at p=0.05 (Table 10). Significant relations were found for sections
A, N(7) , and R. For these sections, the predictive power of OJA is promising. In these cases OJA explained
68%-76% of JVS variance. For other countries the slope coefficient was not statistically significant.

Figure 11: Regression results between OJA and JVS for individual economic activities

4.2.3 Panel data regressions

We apply a panel data regression for JVS as a function of OJA across countries. Such an approach gives
us a holistic overview of the analysed relations. It shows us whether OJA data, as a whole, can be a good
predictor of JVS data, providing that we account only for simple country differences. Starting with fixed
effects model (the within transformation), we follow with testing the random effects (Table 11). The
regressions are based on untransformed data. Alternatively, we used the logarithmic transformation of

(7) OJA data shows strong results for section N – Administrative and support service activities. One must take into account though
that significant part of job ads within this section is published by sub-section of Employment activities by recruitment agencies
that do it on behalf of employers from other sectors. In Skills OVATE, Cedefop treats this NACE sub-section as a separate sector.

Inferring job vacancies from online job advertisements

39

Comparing OJA data with job vacancy statistics

4

Table 10: Summary of regression results for individual economic activities

Intercept
21681** (6.87)
A
2966** (8.49)
B
316778** (6.96)
C
6502** (8.02)
D
13062*** (30.29)
E
226570*** (9.92)
F
260122*** (10.38)
G
157442*** (22.72)
H
113380** (5.62)
I
105999** (7.56)
J
30606** (4.89)
K
15351* (4.29)
L
168996** (7.51)
M
317115*** (23.65)
N
86157*** (19.64)
O
P
59974** (5.22)
Q 276337*** (16.02)
13223* (4.11)
R
44514** (7.16)
S

OJA
0.56* (3.57)
0.24 (0.51)
0.05 (0.89)
0.019 (1.28)
0.041 (1.19)
0.071 (0.47)
0.05 (0.67)
0.012 (0.41)
0.239 (1.5)
0.004 (0.1)
0.067 (1.37)
0.193 (1.76)
0.012 (0.44)
0.042* (2.89)
-0.016 (-0.66)
0.088 (1.3)
0.081 (1.12)
0.365* (3.00)
0.038 (0.6)

R2
0.76
0.06
0.16
0.29
0.26
0.05
0.10
0.04
0.36
0.00
0.32
0.44
0.05
0.68
0.10
0.30
0.24
0.69
0.08

DW test
1.5
0.7*
0.8*
0.9
1.5
0.6*
1.3
3
2.1
2.9
2.9
0.5
2.7
1.7
1.3
1.8
2
1.7
2.6

the data. In this case the goodness-of-fit was even lower, with negligible slope coefficient. We proceed
with untransformed data.

Table 11: Panel data regression between OJA and JVS - countries

Intercept
OJA
Time dummies
Country dummies
R2
DW test
F / Chi2 statistic
Hausman

Within

-
0.013 (1.35)
YES
YES
0.02
2.7
1.8
-

Random
130680*** (3.98)
0.091*** (4.78)
YES
YES
0.20
1.1***
22.8***
22.5***

First differences
1637 (0.51)
0.004 (0.44)
NO
YES
0.002
2.7
0.2
-

Note: t-statistic in (). *** p<0.001, ** p<0.01, * p<0.05. Balanced Panel: n = 16, T = 6, N = 96.

Hausman test results show that differences between countries are consistent, and the fixed effects rather
than random effects can be used to describe the relation between OJA and JVS. However, the relation
between OJA and JVS is weak and statistically insignificant. It is far from one-to-one. As the model shows,
100 online job ads are associated with 1 vacancy. Time effects (Figure 12) show stable relation between
OJA and JVS in the period 2018Q4-2019Q4. However, 2018Q3 significantly differs from the rest.

The relation between first differences of OJA and JVS might be potentially promising, given that online
job ads might not represent the number of vacancies, but may be a good predictor of their changes. First
differences transformation of the data also emphasizes the short-run relations between our variables.
Despite this, within our limited time span the model shows statistical insignificance of such predictions.

We apply regression with increasing window of quarters (starting with two quarters, ending with 6 quar-
ters) to check how the coefficients change when new information is added. Figure 13 presents the
changes in the slope coefficient (marginal effect of OJA). For 4-5 quarters the relation between OJA and
JVS is stable. Adding the sixth quarter decreases the coefficient approximately three times. Thus, we
cannot say that online job ads is a stable predictor of job vacancies.

Inferring job vacancies from online job advertisements

40

Comparing OJA data with job vacancy statistics

4

Figure 12: Time effects for the fixed effects regression between OJA and JVS

Figure 13: Fixed effects regression between OJA and JVS with increasing time span

Note: Results based on balanced panel data regression: n = 16, T = 6, N = 96. Regression contains
individual effects, but not time effects. Horizontal axis represents number of quarters included in the
regression, with 2 meaning 2018Q3-2018Q4 and 7 meaning 2018Q3-2019Q4.

Individual country effects, similarly to the slope coefficient, were not stable when we increased time span
of the analysis (Figure 14). There are countries for which they remained quite stable after the initial ad-
justments. Those countries included mainly Estonia, Poland, and Sweden. For Latvia, the effects tended
to a stable value, and consecutive increments were getting smaller. The most unstable values were esti-
mated for Hungary, Romania, Slovakia, and Slovenia. For other countries fixed effects were moderately

Inferring job vacancies from online job advertisements

41

Comparing OJA data with job vacancy statistics

4

stable.

Figure 14: Individual country effects in fixed effects regression between OJA and JVS with increas-
ing time span

Note: Results based on balanced panel data regression: n = 16, T = 6, N = 96. Regression contains
individual effects, but not time effects. Horizontal axis represents number of quarters included in the
regression, with 2 meaning 2018Q3-2018Q4 and 7 meaning 2018Q3-2019Q4.

Next, we look at the relations between OJA and JVS with panel data regression across industries. Starting
with fixed effects model (the within transformation), we follow with testing the random effects (Table 12).
Likewise the countries, the regressions are based on untransformed data, which give a better fit to the
data.

Table 12: Panel data regression between OJA and JVS - NACE sections

Intercept
OJA
Time dummies
Country dummies
R2
DW test
F / Chi2 statistic
Hausman

Within

-
0.038** (3.39)
YES
YES
0.11
1.6*
11.5**
-

Random
118270*** (5.98)
0.045*** (4.11)
YES
YES
0.13
1.3***
19.2***
7.9**

First differences
142 (1.34)
0.015* (2.22)
NO
YES
0.05
2.3
5.0*
-

Note: t-statistic in (). *** p<0.001, ** p<0.01, * p<0.05. Balanced Panel: n = 19, T = 7, N = 133.

The fixed effects estimator is consistent and better describes the relation between OJA and JVS than
the random effects estimator. The relation is stronger than for countries and statistically significant. 100
online job ads are associated with 4 vacancies. The R2 = 11%. Time effects (Figure 15) for this model
are similarly unstable as the ones in the model with countries.

The first differences model shows the short-run relation between OJA and JVS. For this reason, the OJA
coefficient is even lower than for the rest of the models. Differencing eliminated autocorrelation of resid-
uals, which was present in the previous models. The relation between OJA and JVS is statistically signif-
icant.

Figure 13 presents how the OJA slope coefficient changes if we increase the window of quarters taken to
regression, also for the NACE section. As for the full-time span, the slope coefficient is higher than the one

Inferring job vacancies from online job advertisements

42

Comparing OJA data with job vacancy statistics

4

Figure 15: Time effects for the fixed effects regression between OJA and JVS

for countries. It is stable for 3-5 quarters analysis, but adding the last quarter increases the coefficient.
Changes in this coefficient are smaller than the one in the country breakdown.

Generally, individual NACE effects show similar pattern if we increase time span for the regression (Figure
16). They tend to be stable after a few quarters. The exception is section C, for which the effect was the
most unstable.

Figure 16: Individual industry effects in fixed effects regression between OJA and JVS with in-
creasing time span

Note: Regression contains individual effects, but not time effects. Horizontal axis represents number of
quarters included in the regression, with 2 meaning 2018Q2-2018Q3 and 7 meaning 2018Q2-2019Q4.

The relations between OJA and JVS, estimated with panel data regression do not show promising re-
sults (Table 13). The results are not statistically significant, showing no statistical relation between both

Inferring job vacancies from online job advertisements

43

Comparing OJA data with job vacancy statistics

4

measures of job vacancies. Hausman test results show that differences between ISCO groups are in-
consistent. Despite this, the slope coefficient in random effects model, as well as in the first differences
model is not significant.

Table 13: Panel data regression between OJA and JVS - ISCO major groups in Hungary

Intercept
OJA
Time dummies
Country dummies
R2
DW test
F / Chi2 statistic
Hausman

Within

-
-0.083 (1.51)
YES
YES
0.05
1.6
2.3
-

Random
9426** (5.61)
-0.075 (1.64)
YES
YES
0.05
1.4*
0.8
0.07

First differences
-260** (2.71)
-0.042 (1.74)
NO
YES
0.07
2.4
3.0
-

Note: t-statistic in (). *** p<0.001, ** p<0.01, * p<0.05. Balanced Panel: n = 9, T = 6, N = 54.

Inferring job vacancies from online job advertisements

44

Results

5

5 Results

5.1 Methods and their assumptions

Due to the limitations found in the OJA data available to this study, discussed in previous chapter, we
decided to use capture-recapture methods that are based on distributional assumptions. We especially
focused on:

• naïve trimming, as a way for dealing with over-coverage, based on number of days from grab_date

to the end of given quarter,

• zero-one-truncated capture-recapture (ZOT CR), which is the equivalent to zero-truncated one-
inflated distributions as proved by Böhning and van der Heijden (2019). We used Poisson, Geo-
metric and Negative-Binomial distribution.

We decided to use capture-recapture methods because these methods are suitable for estimating total
number of job vacancies based on limited number of data sources. In order to prepare data for ZOT CR
we took the following steps:

1. we treat the whole OJA database as one data source,

2. for each quarter we calculated the number of job vacancies (based on variable general_id)
present in one, two, three etc. input data sources (variable source). Description of variables can
be found in the Appendix in Table 16.

3. based on results from Table 5 we decided to discard the following countries due to low number
of records in the OJA dataset: Cyprus, Croatia, Estonia, Greece, Latvia, Lithuania, Malta, and
Slovenia.

Furthermore, due to the high differences in the number of job advertisements we decided to divide the
OJA data into two groups in order to apply different adjustments:

1. Group 1 – Bulgaria, Czechia, Ireland, Portugal and Romania – countries with number of advertise-

ments lower than 200 thousand (200k).

2. Group 2 – Belgium, Spain, Germany, the Netherlands and Poland – countries with number of ad-

vertisements higher than 200k.

One of the main challenges that occur in OJA data is the lack of knowledge about the over-coverage error
due to false or outdated advertisements. Figure 17 presents the distribution of the number of days to
the end of the quarter calculated using variable grab_date. Seasonal patterns resemble data collection

Inferring job vacancies from online job advertisements

45

Results

5

process. There are also increases in the number of advertisements scraped at specific dates in 2019Q1,
2019Q3 and 2019Q4.

Certainly, some (unknown) over-coverage is present, as there are advertisements that had grab_date
over 60 or even 100 days. It is reasonable to specify some cut-off threshold to remove outdated adver-
tisements from the study population.

Figure 17: Distribution of number of advertisements by quarters and days to the end of given
quarter based on grab_date variable

Based in the information on the number of days we decided to apply different trimming for the two
groups of countries (under 200k and over 200k) defined above.

Inferring job vacancies from online job advertisements

46

Results

5

5.2 Naïve trimming

The first method that we applied is naïve trimming, meaning that we just specified a cut-off threshold for
number of days from the grab_date to the end of given quarter. For the group of countries with larger
number of job advertisements we used 7 to 30 days (by one day) and for the other group of countries
we used the range of 30 to 60 by 5 days.

Figure 18 presents the result of the trimming procedure and its impact on the number of job advertise-
ments in the OJA data. The red line indicates the number of vacancies according to job vacancy statistics
(seasonally unadjusted), the solid black line shows the number of job vacancies in the OJA data without
any trimming and the gray lines show the transformed data for different alpha-transparency scale.

Figure 18: Comparison of estimates based on JVS (solid red line; seasonally unadjusted data), OJA
data without trimming (solid black line) and trimmed data (solid gray lines)

The main conclusions from this plot are: 1) for Hungary and Czechia the number of vacancies according
to the OJA data are underestimated. In Czechia, JVS are compiled based on administrative data registers
of the Labour Office of Czechia and in Hungary they are based on the Quarterly Establishment Survey; 2)
Ireland is a special case where JVS is significantly smaller than in other countries; 3) JVS are less variable
in comparison to OJA data. A lack of trend is visible in particular for Belgium, Germany or Netherlands;
4) Trimming makes the OJA data less variable but we cannot say anything about seasonal aspect as the
time period is short; 5) in most cases trimming makes the data more similar to JVS but there is no clear
pattern and 6) results of trimming are connected to the number of job vacancies collected in a given
quarter.

The last point from the above discussion is presented in Figure 19 where we calculated absolute relative
difference between trimmed OJA data and estimates from job vacancy statistics. In particular, the figure
presents the following measure

Inferring job vacancies from online job advertisements

47

Results

5

δt
c,q =

| θOJA,t
c,q

− θJVS
c,q

|

θJVS
c,q

× 100%,

(51)

where c represents the country, q represents the quarter, t represents the trimming threshold (i.e. 10
days to the end of given quarter), θOJA,t
is the number of job vacancies for a given country, in a given
c,q
quarter, for given trimming threshold based on OJA data, and finally θJVS

c,q is an estimate based on JVS.

Figure 19 presents δt
c,q for each quarter separately to verify if the same threshold may be applied for all
quarters.
In this plot we set the threshold for grab_date to be between 7 and 50 days before. Coun-
tries significantly vary between countries and quarters. There are no clear patterns as regards to which
threshold should be applied. For instance, Spain and Poland have different threshold for each quarter,
while for some quarters in Germany, Luxembourg, Lithuania, or Finland the same threshold gives similar
results.

Figure 19: Comparison of absolute difference of estimates between OJA trimmed estimates with
JVS for each quarter

To sum up the findings regarding naïve trimming:

• countries significantly vary in patterns of OJA vacancies and JVS,

• it is difficult to set one threshold for all countries,

• setting the same threshold for each quarter in one country is also inconclusive.

Inferring job vacancies from online job advertisements

48

Results

5

5.3 Zero-one truncated capture-recapture results

5.3.1 Estimation procedure

This section covers the estimation of the size of the job vacancies population in selected countries based
on zero-truncated one-inflated (ZTOI) or zero-one truncated (ZOT) capture-recapture approach. In this
chapter, we followed Böhning and van der Heijden (2019) findings regarding equivalence of ZTOI and
ZOT distributions. Under this assumption a Horvitz-Thompson estimator for the target population with
no extra-singletons (i.e. one-inflation) is given by the equation (52)

ˆNnes =

n1
1 − p (x0, θ) − p (x1, θ)

,

(52)

where p(x0, θ), p(x1, θ) are probability of occurrence of 0 and 1 respectively. ˆNnes is an unbiased estima-
tor of the population size with no extra singletons. Hence, Böhning and van der Heijden (2019) construct
an estimator of the hidden units f0 as

ˆf0 = p (x0, θ)

n1
1 − p (x0, θ) − p (x1, θ)

.

(53)

Finally, to achieve the Horvitz-Thompson estimator of the target population of interest, i.e. under ZTOI/ZOT
distribution, as

ˆN = ˆf0 + f1 + n1 = p (x0, θ)

n1
1 − p (x0, θ) − p (x1, θ)

+ f1 + n1,

(54)

where ˆf0 is the estimated hidden population, f1 is the number of units observed one time and n1 is the
rest. ˆN is unbiased estimator of the population size and as θ is unknown and need to be estimated; we
can replace θ with maximum likelihood estimator ˆθ under p+1(x, θ) that leads to:

ˆN = ˆf0 + f1 + n1 = p

(cid:16)

(cid:17)
x0, ˆθ

(cid:16)

n1
(cid:17)
x0, ˆθ

− p

(cid:16)

1 − p

(cid:17) + f1 + n1.

x1, ˆθ

(55)

To estimate θ using maximum likelihood method we define likelihood function for zero-one truncated
distribution as

L++ =

mY

i=2

p++ (xi, θ)fi ,

(56)

where p++ is defined as

p++(x, θ) = p+(x, θ)/ [1 − p+ (x1, θ)] = p(x, θ)/ [1 − p (x0, θ) − p (x1, θ)] ,

(57)

and fi denotes frequency counts for i-th number. In practice, instead of using the likelihood function
defined in (56) we used the log-likelihood given by

mX

log L++ =

fi log p++ (xi, θ) ,

i=2

Inferring job vacancies from online job advertisements

(58)

49

Results

5

where notation is the same as previously.

In order to provide a measure of uncertainty we use non-parametric bootstrap to estimate standard
errors and follow recommendation provided by Böhning and van der Heijden (2019). The procedure to
estimate estimated variance given by

• Draw a sample of size || ˆN || from the observed distribution defined by the probabilities

ˆf0
N ,

ˆf1
N ,

ˆf2
N , ...,

ˆfm
N .

• Derive ˆθ and ˆN for the boostrap sample in 1).

• Repeat step 1) and 2) B times, leading to a sample of estimates N (1), ..., N (B).

• Calculate the bootstrap standard errors as
v
u
u
t 1
B

SE∗ =

BX

(cid:0)

N (b) − ¯N ∗

(cid:1)2

,

b=1

where ¯N ∗ = 1
B

P

B
b=1 N (b) and ,

• calculate relative standard error (i.e. coefficient of variation; CV) by

CV ∗ =

SE∗
¯N ∗

× 100%.

5.3.2 Application

We calculated multiple models, in particular:

• We assumed Poisson

p(x; θ) = p(x; λ) = Pr(X = k) =

λke−λ
k!

Geometric

p(x; θ) = p(x; p) = Pr(X = k) = (1 − p)k−1p.

• We calculated models separately for each quarter (6 quarters),

• We calculated models separately for two groups of countries:

(59)

(60)

(61)

(62)

◦ Under 200k vacancies in quarter – Czechia, Ireland, Bulgaria, Portugal, and Romania (5 coun-

tries) with trimming thresholds (30, 35, 40, 45, 50, 55 and 60 days; 7 thresholds),

◦ Over 200k vacancies in quarter – Belgium, Spain, Germany, the Netherlands, and Poland (5

countries) with trimming thresholds (from 7 to 30 days; 24 thresholds).

This gave 2 × 6 × (5 × 7 + 5 × 24) = 1860 models.

Unfortunately, for the following countries the optimization procedures were not able to find a solu-
tion: Bulgaria, Slovakia, Lithuania, Hungary, Sweden, Luxembourg, and Finland. The main reason is not
enough vacancies observed two or more times.

Within each group we selected the most appropriate distribution (Poisson, Geometric or Negative-Binomial)
based on AIC criterion and estimated the size of the population. The AIC and other characteristics of the
estimated models are reported in Appendix 7.
In this exercise, we used trimming for accounting for
over-coverage and as a sensitivity analysis to different thresholds.

Inferring job vacancies from online job advertisements

50

Results

5

5.3.3 Countries with under 200k advertisements

Figure 20 presents the comparison of estimates based on JVS (red dashed line), OJA data without any
trimming (blue dashed-line) and results from the capture-recapture procedure using zero-one truncated
Poisson distribution. On X axis we have the number of days to the end of given quarter (e.g. 30 days
means that we used vacancies that are max 30 days old). Each panel is defined by country and quarter.

Based on the results we see that in general, untrimmed OJA data for some countries (ie.g. Bulgaria, Por-
tugal, Romania) provide similar results. For Czechia, the difference between JVS and OJA data decreases
as time goes on.

If we look at the results of the estimation procedure, we can see that the most problematic quarters are
the first (2018Q3) and the last (2019Q3). We see variability in estimates as trimming threshold increases.
For Ireland and Portugal trimming made the estimates closer to the JVS (for selected quarters). In general,
there is no clear pattern of estimates and we see that results are sensitive to different trimming.

Figure 20: Comparison of estimates based on JVS (red dashed line), OJA data without trimming
(blue dashed line) and zero-one-truncated capture-recapture size estimator under different trim-
ming dates (30 to 50 by 5)

To compare numerically, we report results for threshold of 40 days in Table 14. The table contains three
variables – JVS, OJA data without trimming and the proposed population size based on zero-one trun-
cated (ZOT) Poisson distribution. This estimate is based on the mean from the bootstrap procedure.

Inferring job vacancies from online job advertisements

51

Results

5

Table 14: Comparison of estimates (in thous.) based on JVS, OJA data without trimming and zero-
one-truncated Poisson capture-recapture size estimator under different trimming to 40 days

country
Czechia

Ireland

Bulgaria

Portugal

Romania

quarter
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3

JVS
282.1
288.5
309.3
303.6
302.4
14.8
12.8
14.3
17.8
13.8
19.3
19.5
22.6
20.3
18.9
29.6
27.4
33.5
34.4
36.5
63.7
58.8
58.2
55.0
55.0

OJA Trimmed
45.7
93.1
33.9
138.4
79.9
264.8
74.3
239.2
74.6
225.0
47.6
105.5
38.7
135.3
58.0
180.3
50.1
161.8
55.9
149.9
8.0
11.9
17.7
35.0
27.7
83.9
20.2
60.7
18.5
49.1
21.5
40.8
26.1
71.5
26.2
79.9
29.2
68.1
30.8
74.4
9.9
17.1
24.2
43.2
30.2
134.3
24.9
80.2
42.9
91.9

ZOT Poisson
1417.6
29.6
121.8
733.1
106.8
41.2
14.5
79.2
171.7
261.5
349.3
227.4
371.2
245.7
763.3
496.7
35.6
80.3
52.5
230.2
137.8
9.6
253.9
422.7
770.2

CV
92.9
10.7
14.3
32.5
44.3
17.0
0.9
3.9
10.4
10.4
3.4
0.4
1.5
4.5
7.0
11.5
0.7
3.1
2.1
7.2
128.4
0.2
27.9
27.4
22.5

Inferring job vacancies from online job advertisements

52

Results

5

5.3.4 Countries over 200k advertisements

Figure 21 presents the comparison of estimates based on JVS (red dashed line), OJA data without any
trimming (blue dashed-line) and results from the capture-recapture procedure using zero-one truncated
Poisson distribution for countries with high number of vacancies. On the X axis we have the number of
days to the end of a given quarter (e.g. 10 days means that we used vacancies that are max 10 days old).
Each panel is defined by country and quarter.

Contrary to the previous results for the group of below 200k, the countries in this group are more stable.
Exceptions are the first quarter (2018Q3) and the last quarter (2019Q3). We also note that for Belgium
and Poland, the number of job vacancies are about two times higher than JVS. For other countries, over-
coverage error is significant as the number of reported vacancies based on OJA data is three or more
times higher. We note that in comparison to countries with a lower number of advertisements, trimming
and usage of ZOT Poisson capture-recapture, is more stable and closer to estimated JVS. This pattern
is clearly visible for Germany, where trimming and ZOT Poisson significantly decreased the number of
vacancies initially estimated from OJA data.

Figure 21: Comparison of estimates based on JVS (red dashed line), OJA data without trimming
(blue dashed line) and zero-one-truncated capture-recapture size estimator under different trim-
ming dates (7 to 30)

To compare numerically, we reported results for threshold of 40 days in Table 15. The table contains
three variables – JVS, OJA data without trimming and the proposed population size based on zero-one
truncated (ZOT) Poisson distribution. This estimate is based on the mean from the bootstrap procedure.

Inferring job vacancies from online job advertisements

53

Results

5

Table 15: Comparison of estimates (in thous.) based on JVS, OJA data without trimming and zero-
one-truncated Poisson capture-recapture size estimator under different trimming to 20 days

country
Belgium

Spain

Germany

Netherlands

Poland

quarter
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3
2018Q3
2018Q4
2019Q1
2019Q2
2019Q3

JVS
149.2
141.7
147.7
139.0
140.9
123.8
123.1
130.3
138.5
135.9
1237.4
1458.4
1380.3
1389.2
1359.4
258.3
254.9
278.9
294.0
283.7
157.2
139.2
142.5
151.8
148.6

OJA Trimmed
70.9
56.2
97.8
85.1
80.3
39.2
57.7
109.3
99.2
142.0
427.5
513.7
546.0
269.2
428.1
151.5
97.1
89.3
65.0
133.6
87.0
34.8
72.0
74.5
98.1

305.8
367.8
599.4
515.1
452.8
226.8
314.1
532.4
511.2
521.7
2356.3
3286.3
4607.9
2618.4
2771.0
574.1
692.3
864.5
463.0
599.6
310.0
320.5
400.9
266.0
403.0

Proposed
678.9
89.7
304.0
1827.9
248.7
73.3
75.3
175.0
293.9
554.2
911.2
637.0
817.9
500.6
1330.4
856.1
118.9
153.3
108.0
404.3
1084.0
37.9
535.0
1003.5
1715.0

CV
42.4
10.2
9.5
19.7
22.9
10.1
4.0
2.7
6.2
8.4
1.9
0.5
1.1
2.9
5.4
6.3
1.6
1.8
2.1
4.7
36.0
0.1
16.2
18.8
16.8

Inferring job vacancies from online job advertisements

54

Results

5

5.4 Conclusions

Based on the results we conclude that:

• Results are inconclusive as one method provides different results for different study countries,

• With unknown quality of OJA data it is not possible to derive unbiased estimates of total number

of job vacancies,

• Audit sample is needed to assess OJA data,

• As the OJA data used in this study significantly differ between countries, application of the same

method leads to different results,

• Results for bigger countries measured by number of vacancies tend to provide less variable esti-

mates, thus application of these methods may be more suitable for such countries,

• Sensitivity analysis of the trimming procedure suggests that there is additional variability due to

over-coverage in OJA data,

• Over-coverage is still a serious problem in OJA data, but it is not possible to verify to what extent

from the current study,

• We consider the OJA data as available from Cedefop at the time of the study more appropriate for

analysing skills rather than estimating the number of job vacancies.

The OJA dataset available from Cedefop that was the subject of the analysis certainly is an interesting
source of information but at the time of the study has a limited usability for estimating the total number
of vacancies. The main limitation is the unknown (statistical) quality of the OJA data. We suggest re-
designing the study that will cover certain disadvantages of the current approach. Another option, but
in our opinion only temporary, is to take a subsample of Cedefop’s data that will be suitable to predict
job vacancies. More detailed conclusions are in the next chapter.

Inferring job vacancies from online job advertisements

55

Findings and recommendations

6

6 Findings and recommenda-

tions

6.1 Main findings

6.1.1 Correlation between OJA and JVS

Analysing the relation between the Online Job Advertisements (OJA) dataset available from Cedefop at
the time of this study and Job Vacancies Statistics (JVS) published by Eurostat, we reached the following
findings:

• The OJA dataset identified 40%-91% job offers more than JVS did in the analysed period.

• The OJA dataset show a lot more variation than JVS.

• The distributions found in the OJA dataset are more leptokurtic and right skewed than analogous

distributions in JVS.

• The relation between both measures of vacancies across time and countries was not statistically
significant, whilst across time and industries it was found to be statistically significant. The latter
relation was relatively stable over time.

• The relations between OJA and JVS differ significantly between countries, industries, and occupa-

tions.

• Satisfactory predictions of JVS based on OJA were made for four countries: Czechia, Latvia, Bul-

garia, and Sweden, and for three NACE sections: A, N, and R.

• For ISCO major occupational groups, but only based on one country (Hungary), we found large

differences between OJA and JVS.

6.1.2 Population size estimation

Based on the estimation analysis we conclude that:

• Results are inconclusive and further studies are needed.

• Unknown quality of the OJA dataset severely limited the possibility to derive unbiased estimates

of total number of job vacancies.

• Audit sample is needed to assess the OJA data.

• As the OJA data significantly differs between countries, the application of the same method to all

the countries may lead to different results.

• Results for bigger countries, measured by number of vacancies, tend to provide less variable esti-

mates, thus application of these methods may be more suitable for such countries.

Inferring job vacancies from online job advertisements

56

Findings and recommendations

6

• Sensitivity analysis of trimming suggest that there is additional variability due to over-coverage in

the OJA data.

• Over-coverage is still a serious problem in the OJA data, but it is not possible to verify to what

extend from the current study.

6.2 Main limitations

Based on our analysis we conclude that in its current form the OJA data available from Cedefop is not
suitable for deriving the total number of job vacancies. At this stage, it should be rather considered as
a source for statistics on skills and further improvements need to be made in order to attempt to make
estimations for the total number of job vacancies.

We identified the following limitations of using the available OJA data from Cedefop for the estimation
of the total number of job vacancies:

1. Date of collection and date of expiration – they are based on web-scraping not actual dates from

the advertisements;

2. Overcoverage – unknown outdated or erroneous ads;

3. Duplicates – there still may be duplicates despite the procedures applied;

4. Overlap – in most cases overlap between data sources is very small;

5. Variability in data collection – data are collected from various websites and with various frequen-

cies. There are high discrepancies between quarters;

We also identified limitations regarding Job Vacancies Statistics conducted by Member-States and the
availability of data at Eurostat’s website:

1. Member-States differ in the methodology of calculating job vacancy statistics – they refer to dif-

ferent populations, use different sources and different estimation methods;

2. Data available at Eurostat’s website is not complete – some countries report all NACE sections,
some only totals, or specific sub-populations. Only Hungary reports information about occupa-
tions;

3.

Information regarding the quality of JVS estimates is outdated (e.g. 2017) and limited to specific
subpopulations – we cannot use these data to properly compare JVS with OJA data;

Further, we faced considerable limitations during the study.

Time series dimension. Having the data for 2018.7-2019.12 (not a complete period of two years) severely
limits the application of time series models and analysis. Such data does not allow for a proper season-
ality analysis. Without this, we cannot properly extract and compare other time series components of
JVS and OJA, and most importantly trends and cycles. At least three years are required to perform such
an analysis. Seasonality patterns in online data might differ from the ones in survey data, which is what
should be analysed when having longer time series. Those differences might be one of this data speci-
ficities and should not lower the usefulness of online data. Longer time series will enable the analysis of
OJA as high frequency data in comparison to quarterly JVS.

NACE section dimension. We have serious doubts about the accuracy of the NACE variable extracted
from the job advertisements and available in the OJA dataset. Moreover, this is the only variable reported
in Eurostat’s JVS and due to low quality in Cedefop, significant differences are encountered. According

Inferring job vacancies from online job advertisements

57

Findings and recommendations

6

to Cedefop’s documentation (Cedefop, 2019c) Annex I: Occupations, skills, and other variables classifica-
tion), NACE section is classified either based on a structured field or searched for in a workplace descrip-
tion. The latter suggests that the NACE section is not based on the NACE of a company that published
a job ad (as it is in official statistics, e.g. JVS), but rather based on a workplace that might potentially be
various. For example, an IT company may want to hire an accountant. This leads to a different treatment
of the NACE section by Cedefop than by Eurostat.

ISCO occupation dimension. While ISCO occupation is present in the OJA dataset, it is not reported
by almost any of the EU countries in Eurostat’s JVS statistics. This also limits the use of Cedefop’s data
on skills. Skills data is potentially a large value added of Cedefop’s OJA data in comparison to official
statistics. However, before using data on skills, it would be important to compare the occupational dis-
tributions of OJA and JVS to analyse representation of occupations in online data. The next step would
be to analyse demand for skills.

Longitudinal data. There are no job offers with the same general_id between different grab_date
and expire_date periods. This limits tracking of the same advertisement in time and performance of
longitudinal analysis (see Summary regarding the data section).

6.3 Recommendations

6.3.1 For the OJA dataset

• Include additional table with links between advertisements and source websites;

• Include information about the quality of the automatic classifiers used to predict the statistical

variables from the content of the job advertisement;

6.3.2 Data sources

We recommend focusing on data sources from Public Employment Offices (PEO) which may be of higher
quality than commercial websites or job ads aggregators. For instance, in the Polish Ministry of the Min-
istry of Family, Labour and Social Policy that supervises PEOs provides online service – the Central Job
Advertisements Database – with all job ads that are registered in PEOs. Each job ad is verified by a clerk
at local PEOs, classified to occupation and all information about the employer is provided. Moreover,
outdated, erroneous or if employee(s) was/were found, ads are removed by PEOs which minimize po-
tential over-coverage error. These data may be further used as a training data for advertisements from
commercial websites both in terms of occupation as in terms of delays in reporting.

Cedefop uses machine learning algorithms to collect, deduplicate, classify, and present job advertise-
ments. However, to obtain measures of estimation errors at the stage of OJA-JVS inference, measures
of accuracy of Cedefop’s algorithms are needed. Another problem is that we do not know the share
of possible obsolete job advertisements, that is job advertisements for job positions that have been al-
ready filled. Companies might not inform timely the job board about filling the job vacancy. Since most
websites introduce a job advertisement publication period, after which this job advertisement is auto-
matically removed from the website (most often 30 days), this bias will last for this period.

6.3.3 Audit sample

Furthermore, to assess the quality of the OJA data, an audit sample should be considered. This approach
is recommended when one is interested in estimating coverage or classification error. Examples regard-
ing this approach can be found in:

• the prevalence of cybercrime in the Netherlands – See The accuracy of estimators based on a binary

classifier,

• correcting misclassification in enterprises – Delden et al. (2016),

Inferring job vacancies from online job advertisements

58

Findings and recommendations

6

• correcting labour force status – Zhang (2005),

• coverage error of register – Zhang (2015)

Having that in mind, the audit sample may be designed for:

1. verification of over-coverage error due to false, outdated or out-of-scope units;

2. verification of over-coverage error due to insufficient deduplication;

3. verification of classification error due to automatic classification error;

We recommend preparing two independent audit samples to tackle both over-coverage errors sepa-
rately (out-of-scope units and duplicated records) and the process should be conducted separately on
each country. Moreover, we suggest that the samples may be drawn from the final OJA dataset. In par-
ticular, we propose the following procedure:

• Let the final OJA dataset be the reference data used to draw a sample. The dataset of size nt may
contain deduplicated job advertisements that (according to the OJA data) are open at day t,
• Draw two independent audit samples – for over-coverage (s1) and for deduplication (s2) according

to the following schema:

1. sample s1

◦ simple random sampling or Poisson sampling proportional to number of days between

placing advertisement and time t,

◦ size of the sample should be manageable for team of people who conduct clerical review
– this requires contacting the company or recruiting agency to verify if the vacancy is in
fact open. For instance, one may consider 1% sample, but it depends on the available
funds,

◦ based on the result of clerical review, estimate the error due to over-coverage that might
be followed by modelling procedure i.e. binary classification procedure where 1 = out-
of-scope and 0 = open vacancy,

2. sample s2

◦ we recommend applying some probabilistic procedure to duplicate the records. Cur-

rently, Cedefop duplicates rows based on comparison of specific fields,

◦ compute distances / probabilities of duplication between pairs
◦ sample pairs that have the lowest distance / highest probability and have the highest

distance / lowest probability,

◦ verify which pairs are in fact duplicates,
◦ conduct supervised record linkage to deduplicate records.

3. Combine results from sample s1 and s2 to verify overlap between out-of-scope units and du-
plicated records. Based on that result, estimate the over-coverage error, and use it to down-
size nt records.

This approach requires that additionally to web-scraping, a survey is conducted in parallel. However,
this may exceed the goal of this study, that is to estimate the number of vacancies solely based on the
existing online data. Nevertheless, it would increase the quality of obtained data.

6.3.4 Additional scraping

We also recommend that Eurostat and Cedefop additionally scrapes, where it is possible, information
about the company that publishes an online job advertisement. It could be classified or linked with the
business register. We are aware that it will not be always possible or the company might differ from the
one that in fact offers a job position (i.e. recruiting agency).

Inferring job vacancies from online job advertisements

59

Findings and recommendations

6

6.4 Conclusion

OJA data are certainly very useful and promising. Cedefop’s OJA dataset is large, it includes many coun-
tries, and many breakdowns of job ads. Among them, there are estimates of individual skills, that can be
connected to occupations. However, for now, the quantitative results of inferring official job vacancies
statistics from Cedefop’s OJA data are poor. The number of job offers differ from vacancies by means,
variation and other central moments of both distributions. This might be due to fast-developing struc-
tures of online job boards, as well as variable interest of job seekers and companies in existing online
job boards, and formation of new job boards. Such changes happen randomly, so they are not easily
predictable. A continuous monitoring of job boards might be supplemented with a validation of job
advertisements with companies’ websites. It might also lead to elimination of possible outdated job of-
fers. Furthermore, we recommend development of a selection method for data sources on online job
ads to account for various types of job offers, e.g. small and large companies, local and country-wide job
ads, low-skilled and high-skilled occupations etc. Currently the composition of job boards considered in
different countries varies. It may be one of the key factors behind varying properties of data on OJA.

The distributions of OJA and JVS are different. The more disaggregate data we use, the bigger are those
differences. We find some promising results for forecasting JVS with aggregate data on OJA for four
countries and three NACE sections. However, these relations need to be analysed when longer time
series are available.

The number of online job offers gathered by Cedefop is 85% higher than the stock of Eurostat’s JVS job
vacancies. Having so many job advertisements creates the opportunity of using only some of them as
predictors for job vacancies. One of the typical solutions, most often used in the literature, is to draw a
subsample of online job offers to ensure comparability of distributions according to specific variables,
out of which country and industry are the most important.

Classification algorithms are based on dictionaries, for example dictionary of occupations and their syn-
onyms. However, some languages contain a lot richer variety of words than English language does, for
example Poland, Slovakia, Czechia, Hungary.
In such countries the dictionary of basic forms of words
might not be enough. We recommend lemmatizing words from dictionaries and job advertisements by
transforming these words to their basic forms, for example verbs to infinitives, nouns to first person, first
case, singular forms. It should improve recognition of phrases from dictionaries in job advertisements.

We also suggest considering OJA data as a source of information about skills, and showing qualitative
data, for example, skills rankings, classifications of skills across occupations, and other characteristics
of job offers, like type of contract etc. Such detailed data are obvious advantages of OJA data over
probability-based surveys. With these data, structural changes in the labour market may be analysed
more carefully than traditional official statistics allow it. Also, despite large variance, OJA data may po-
tentially show leading properties in comparison to other labour market aggregates. They may possibly
lead the business cycle turning points, especially as OJA data are high frequency, high granularity, and
may be gathered, analysed, and published with a small lag in comparison to job vacancy statistics. These
functions should be further explored with longer OJA data time series, and more accurate data from
particular EU Member-States’ statistical offices, especially for the NUTS2 regions and ISCO occupational
breakdowns.

Inferring job vacancies from online job advertisements

60

References

7 References

References

Abraham, K. G. and Wachter, M. (1987). Help-wanted advertising, job vacancies, and unemployment.
Brookings papers on economic activity, 1987(1):207–248.

Azar, J. A., Marinescu, I., Steinbaum, M. I., and Taska, B. (2018). Concentration in us labor markets: Evidence
from online vacancy data. Technical report, National Bureau of Economic Research.

Barnichon, R. (2010). Building a composite help-wanted index. Economics Letters, 109(3):175–178.

Böhning, D. and van der Heijden, P. G. (2019). The identity of the zero-truncated, one-inflated likelihood
and the zero-one-truncated likelihood for general count densities with an application to drink-driving in
britain. The Annals of Applied Statistics, 13(2):1198–1211.

Bohning, D., Van der Heijden, P. G., and Bunge, J. (2017). Capture-recapture methods for the social and
medical sciences. CRC Press.

Bürkner, P.-C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical
Software, 80(1):1–28.

Cajner, T., Ratner, D., et al. (2016). A cautionary note on the help wanted online data. FEDS Notes,
Board of Governors of the Federal Reserve System https://www. federalreserve. gov/econresdata/notes/feds-
notes/2016/acautionary-note-on-the-help-wanted-online-data-20160623. html.

Carnevale, A. P., Jayasundera, T., and Repnikov, D. (2014). Understanding online job ads data. Georgetown
University, Center on Education and the Workforce, Technical Report (April).

Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P.,
and Riddell, A. (2017). Stan: A probabilistic programming language. Journal of statistical software, 76(1).

Cedefop (2019a). Online job vacancies and skills analysis: a Cedefop pan-european approach.

Cedefop (2019b). The online job vacancy market in the eu: driving forces and emerging trends. Cedefop
research paper, 72.

Cedefop (2019c). Project “real-time labour market information on skill requirements: feasibility study
and working prototype”. final report.

Chang, T. and Kott, P. S. (2008). Using calibration weighting to adjust for nonresponse under a plausible
model. Biometrika, 95(3):555–571.

Chatterjee, K. and Bhuyan, P. (2017). A New Capture-Recapture Model in Dual-record System A New
Capture-Recapture Model in Dual-record System.

Chatterjee, K. and Bhuyan, P. (2019a). On the estimation of population size from a dependent triple-
record system. Journal of the Royal Statistical Society. Series A: Statistics in Society, 182(4):1487–1501.

Inferring job vacancies from online job advertisements

61

References

Chatterjee, K. and Bhuyan, P. (2019b). On the estimation of population size from a post- stratified two-
sample capture – recapture data under dependence. Journal of Statistical Computation and Simulation
ISSN:.

Chatterjee, K. and Mukherjee, D. (2018). A new integrated likelihood for estimating population size in
dependent dual-record system. Canadian Journal of Statistics, 46(4):577–592.

Colombo, E., Mercorio, F., and Mezzanzanica, M. (2019). Ai meets labor market: exploring the link be-
tween automation and skills. Information Economics and Policy.

Daniel, C., Silva, G., and Alberto, J. (2005). Bayesian analysis of correlated misclassified binary data. Com-
putational Statistics & Data Analysis, 49:1120–1131.

Davis, S. J., Faberman, R. J., and Haltiwanger, J. C. (2013). The establishment-level behavior of vacancies
and hiring. The Quarterly Journal of Economics, 128(2):581–622.

Delden, A., Scholtus, S., and Burger, J. (2016). Accuracy of mixed-source statistics as affected by classifi-
cation errors. Journal of Official Statistics, 32:619–642.

Deming, D. and Kahn, L. B. (2018). Skill requirements across firms and labor markets: Evidence from job
postings for professionals. Journal of Labor Economics, 36(S1):S337–S369.

Deville, J. and Lavallée, P. (2006).
method. Survey Methodology, 32(2):165.

Indirect sampling: The foundations of the generalized weight share

Dunlop, J. T. (1966). Job vacancy measures and economic analysis. In The measurement and interpretation
of job vacancies, pages 27–47. NBER.

Fader, P. S., Hardie, B. G., and Lee, K. L. (2005). “counting your customers” the easy way: An alternative to
the pareto/nbd model. Marketing science, 24(2):275–284.

Godwin, R. T. and Böhning, D. (2017). Estimation of the population size by using the one-inflated positive
poisson model. Journal of the Royal Statistical Society: Series C (Applied Statistics), 66(2):425–448.

Hershbein, B. and Kahn, L. B. (2018). Do recessions accelerate routine-biased technological change?
evidence from vacancy postings. American Economic Review, 108(7):1737–72.

Kott, P. S. and Chang, T. (2010). Using calibration weighting to adjust for nonignorable unit nonresponse.
Journal of the American Statistical Association, 105(491):1265–1275.

Kuhn, P. and Skuterud, M. (2004). Internet job search and unemployment durations. American Economic
Review, 94(1):218–232.

Lavallée, P. (2009). Indirect sampling. Springer Science & Business Media.

Liu, H. and Zhang, Z. (2017). Logistic regression with misclassification in binary outcome variables : a
method and software. Behaviormetrika, 44(2):447–476.

Lohr, S. L. and Brick, J. M. (2012). Blending domain estimates from two victimization surveys with possible
bias. Canadian Journal of Statistics, 40(4):679–696.

Marinescu, I. and Rathelot, R. (2018). Mismatch unemployment and the geography of job search. Amer-
ican Economic Journal: Macroeconomics, 10(3):42–70.

Marinescu, I. and Wolthoff, R. (2016). Opening the black box of the matching function: The power of
words. Technical report, National Bureau of Economic Research.

Meyer, B. D. and Mittag, N. (2017). Misclassification in binary choice models. Journal of Econometrics,
200(2):295–311.

Oliveira, G. L. D., Loschi, R. H., and Assunção, R. M. (2017). A random-censoring Poisson model for under-
reported data. (June):4873–4892.

Inferring job vacancies from online job advertisements

62

References

Pater, R. (2017). Is there a beveridge curve in the short and the long run? Journal of applied economics,
20(2):283–303.

Pater, R., Szkola, J., and Kozak, M. (2019). A method for measuring detailed demand for workers’ compe-
tences. Economics: The Open-Access, Open-Assessment E-Journal, 13(2019-27):1–30.

Pfeffermann, D. and Sikov, A. (2011).
household surveys with missing covariate information. Journal of Official Statistics, 27(2):181–209.

Imputation and estimation under nonignorable nonresponse in

Pires, M. C. and Quinino, R. d. C. (2019). Repeated responses in misclassification binary regression: A
bayesian approach. Statistical Modelling, 19(4):412–443.

R Core Team (2019). R: A Language and Environment for Statistical Computing. R Foundation for Statistical
Computing, Vienna, Austria.

Riddles, M. K., Kim, J. K., and Im, J. (2016). A propensity-score-adjustment method for nonignorable
nonresponse. Journal of Survey Statistics and Methodology, 4(2):215–245.

Rothwell, J. (2014). Still searching: Job vacancies and stem skills. Report. Washington: Brookings Institu-
tion.

Roy, S., Banerjee, T., and Maiti, T. (2005). Measurement error model for misclassified binary responses.
Statistics in Medicine, 24:269–283.

Roy, S., Das, K., and Sarkar, A. (2013). Analysis of binary data with the possibility of wrong ascertainment.
Statistica Neerlandica, 67(3):293–310.

Schmittlein, D. C., Morrison, D. G., and Colombo, R. (1987). Counting your customers: Who-are they and
what will they do next? Management science, 33(1):1–24.

Sikov, A. (2018). A brief reivew of approaches to non-ignorable non-response.
Review, 86(3):415–441.

International Statistical

Stoner, O., Economou, T., Drummond, G., Stoner, O., Economou, T., Drummond, G., Stoner, O., Economou,
T., and Drummond, G. (2019). A Hierarchical Framework for Correcting Under- Reporting in Count Data A
Hierarchical Framework for Correcting Under-Reporting in Count Data. Journal of the American Statistical
Association, 0(0):1–17.

Tang, N. and Ju, Y. (2018). Statistical inference for nonignorable missing-data problems: a selective re-
view. Statistical Theory and Related Fields, 2(2):105–133.

The Conference Board (2018). The conference board help wanted online® technical note. Technical
report.

Turrell, A., Speigner, B., Djumalieva, J., Copple, D., and Thurgood, J. (2018). Using job vacancies to under-
stand the effects of labour market mismatch on uk output and productivity.

Turrell, A., Speigner, B. J., Djumalieva, J., Copple, D., and Thurgood, J. (2019). Transforming naturally
occurring text data into economic statistics: The case of online job vacancy postings. Technical report,
National Bureau of Economic Research.

Zhang, L.-C. (2005). On the bias in gross labour flow estimates due to nonresponse and misclassification.
Journal of Official Statistics, 21.

Zhang, L.-C. (2008). Developing methods for determining the number of unauthorized foreigners in
norway. Statistisk Sentralbyrå/Utlendingsdirektoratet, Oslo Garcia, Jose Miguel Morales.

Zhang, L.-c. (2015). On Modelling Register Coverage Errors. 31(3):381–396.

Zhang, L.-C. and Chambers, R. L. (2019). Analysis of Integrated Data. CRC Press.

Zhang, L.-C. and Dunne, J. (2017). Trimmed dual system estimation.
the Social and Medical Sciences, pages 237–257. Chapman and Hall/CRC.

In Capture-Recapture Methods for

Inferring job vacancies from online job advertisements

63

Annex

Annex

Description of Cedefop’s OJA dataset

We got access to the database (cedefop_datalab.ft_advertisement_en and cedefop_datalab.ft-
_skill_analysis_en) with the following variables

Table 16: Description of variables in Cedefop’s OJA dataset (ft_advertisement_en and
ft_skill_analysis_en)

Variable
general_id
grab_date
expire_date
lang
idesco_level_4,3,2,1
esco_level_4,3,2,1
idescoskill_level_3,2,1 (ft_skill_analysis_en)
escoskill_level_3,2,1 (ft_skill_analysis_en)
idcity
city
idprovince
province
idregion
region
idmacro_region
macro_region
idcountry
country
idcontract
contract
ideducational_level
educational_level
idsector
sector
idmacro_sector
macro_sector
idcategory_sector
category_sector
idsalary
salary
idworking_hours
working_hours
idexperience
experience
source_category
sourcecountry
source
site
companyname

Description
Ad identifier
First date of web-scrapping
Last date of web-scrapping or expire date
Language
ID of ESCO level 4,3,2,1
Name of ECSCO level 4,3,2,1
ID of ESCO skills level 3,2,1
Name of ESCO skills 3,2,1
City ID
City name
Province ID
Province name
Region ID
Region name
Macroregion ID
Macroregion name
Country ID
Country name
Type of contract ID
Type of contract name
Educational level ID
Educational level name
Sector ID
Sector name
Macrosector ID
Macrosector Name
Category sector ID
Category sector name
Salary ID
Salary name
Working hours ID
Working hours name
Experience years ID
Experience years name
Data source (website)
Source country
Type of source
Exact name of the website
Company name

Inferring job vacancies from online job advertisements

64

Detailed results of models

ZERO-ONE TRUNCATED CAPTURE-RECAPTURE METHODS

Annex

Figure 22: Comparison of CV of estimates based on JVS (red dashed line), OJA data without trim-
ming (blue dashed line) and zero-one-truncated capture-recapture size estimator under different
trimming dates (7 to 30)

Table 17: Summary statistics for information criterion for big countries

Country
Belgium
Spain
Germany
Netherlands
Poland

# Poisson better
74
108
70
117
98

# Geometric better Avg AIC Poisson Avg AIC Geometric
721.73
2600.42
14322.83
6601.33
1255.11

720.62
2588.86
14279.85
6564.95
1230.90

46
12
50
3
22

Note: # Poisson better denotes how many times the Poisson model was better, # Geometric better
denotes how many times the Geometric distribution was better, Avg AIC Poisson and Avg AIC Geometric
denotes average AIC criterion for Poisson and Geometric distribution, respectively.

Inferring job vacancies from online job advertisements

65

Annex

Figure 23: Comparison of CV of estimates based on JVS (red dashed line), OJA data without trim-
ming (blue dashed line) and zero-one-truncated capture-recapture size estimator under different
trimming dates (30 to 60 by 5)

Table 18: Summary statistics for information criterion for small countries

Country
Czechia
Ireland
Bulgaria
Portugal
Romania

# Poisson better
20
37
24
37
32

# Geometric better Avg AIC Poisson Avg AIC Geometric
429.55
1594.15
9225.69
4122.33
871.23

429.05
1588.00
9202.68
4097.21
851.73

20
3
16
3
8

Note: # Poisson better denotes how many times the Poisson model was better, # Geometric better
denotes how many times the Geometric distribution was better, Avg AIC Poisson and Avg AIC Geometric
denotes average AIC criterion for Poisson and Geometric distribution, respectively.

Inferring job vacancies from online job advertisements

66

Annex

R codes for estimation of total number of job vacancies

ZERO-ONE TRUNCATED CAPTURE-RECAPTURE MODELS

1. Log-likelihood of zero-one truncated Poisson distribution

ll_trun_pois <- function(x, par) {
vec <- as.vector(table(x))
lev <- as.numeric(names(table(x)))
den <- 1 - dpois(x = 0, lambda = par) - dpois(x = 1, lambda = par)
probs <- dpois(x = lev, lambda = par)
ll <- vec * log(probs / den)
sum(ll)

}

2. Log-likelihood of zero-one truncated Geometric distribution

ll_trun_geom <- function(x, par) {
vec <- as.vector(table(x))
lev <- as.numeric(names(table(x)))
den <- 1 - dgeom(x = 0, prob = par) - dgeom(x = 1, prob = par)
probs <- dgeom(x = lev, prob = par)
ll <- vec * log(probs / den)
sum(ll)

}

4. Estimation using maxLik package

est_pois <- maxLik(logLik = ll_trun_pois,
start = 0.1, method = "NR", x = data)
est_geom <- maxLik(logLik = ll_trun_geom,
start = 0.1, method = "NR", x = data)

5. Estimation of population sizes

popsizes <- c(

poisson = dpois(0, coef(est_pois))*sum(res$times[-1])/
(1 - dpois(0, coef(est_pois)) - dpois(1, coef(est_pois))) +
sum(res$times),

geom = dgeom(0, coef(est_geom))*sum(res$times[-1])/
(1 - dgeom(0, coef(est_geom)) - dgeom(1, coef(est_geom))) +
sum(res$times))

)

6. Bootstrap for Poisson distribution

pop_size <- popsizes[1]

Inferring job vacancies from online job advertisements

67

Annex

probs <- c(pop_size- sum(res$times), res$times)/pop_size
boots <- 200
sampled_freqs <- rmultinom(n = boots, size = pop_size, prob = probs)
sampled_freqs <- t(sampled_freqs)
pop_boot_est <- numeric(boots)

for (b in 1:boots) {

yyy_boot <- rep(res$N[-1], sampled_freqs[b,-c(1,2)])
est_boot <- maxLik(logLik = ll_trun_pois,
start = coef(est_pois), method = "NR", x = yyy_boot)
pop_boot_est[b] <- dpois(0, coef(est_boot))*sum(res$times[-1])/
(1 - dpois(0, coef(est_boot)) - dpois(1, coef(est_boot))) + sum(res$times)

}

Inferring job vacancies from online job advertisements

68

GETTING IN TOUCH WITH THE EU 

In person 
All over the European Union there are hundreds of Europe Direct information centres. You can find the 
address of the centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 
Europe Direct is a service that answers your questions about the European Union. You can contact this 
service: 
– by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 
– at the following standard number: +32 22999696 or  
– by email via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 
Information about the European Union in all the official languages of the EU is available on the Europa 
website at: https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications at: https://op.europa.eu/en/publications. 
Multiple copies of free publications may be obtained by contacting Europe Direct or your local information 
centre (see https://europa.eu/european-union/contact_en). 

EU law and related documents 
For access to legal information from the EU, including all EU law since 1952 in all the official language 
versions, go to EUR-Lex at: http://eur-lex.europa.eu 

Open data from the EU 
The EU Open Data Portal (http://data.europa.eu/euodp/en) provides access to datasets from the EU. 
Data can be downloaded and reused for free, for both commercial and non-commercial purposes. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
P
D
F

:

K
S
-
T
C
-
2
0
-
0
0
8
-
E
N
N

-

Inferring job vacancies from  

online job advertisements?

This study analyses online job advertisements collected by Cedefop in a 
pan-European approach. The main objective of the study is to develop 
estimator(s) for the number of job vacancies from data on online job 
advertisements taking into account the differences in the statistical unit 
and coverage. Although there are promising results for certain countries 
and industries, at the present stage the data are still experimental and 
could not be used to estimate the total number of vacancies. Based on 
the results, recommendations are provided to overcome the current 
limitations for future analysis..

For more information
https://ec.europa.eu/eurostat/

PDF: ISBN  978-92-76-23299-5

Title 2020 edition