Var3n/jobads-identification

JobAds Identification Model

The JobAds Identification Model identifies which text segments in historical German-language newspapers (1850-1950) are job advertisements. It is an ELECTRA-based model (zeitungs-lm-v1, Schweter, 2024) fine-tuned by us for binary classification on raw OCR output. On the held-out evaluation data it reached an area under the precision-recall curve (AUC-PR) of 0.969.

Intended use

The model classifies a single text segment (an OCR text region of a newspaper page) as either a job advertisement or not. It is intended as a first filtering step to find job advertisements in large collections of digitised historical newspapers.

• Language: German
• Period: trained for newspapers from 1850 to 1950
• Input: raw, uncorrected OCR text of one text region
• Output: two classes

Label	Meaning
0	other (not a job advertisement)
1	job advertisement

 

Important: the model was trained and is meant to be applied to raw OCR text (here: OCR with the frak2021 model).

What counts as a job advertisement

A segment is labelled 1 if it falls into any of these categories:

• Headings: headings above or within job advertisement sections, such as Offene Stelle ("Open positions").
• Job offers: an open position is being advertised.
• Job searches: someone is looking for a position.
• Service offers: a service is offered, e.g. language lessons, without creating a full-time work relationship.
• Mediations: positions are offered or sought through a mediator, e.g. an agency.

Use with the job ad category classifier

The model can be combined with a subsequent model that assigns the identified job ads to one of the categories above (heading, job offer, job search, service offer, mediation). That model is not yet published; a separate model card and a link will follow.

Out-of-scope use

The model has not been validated for newspapers outside 1850-1950, for languages other than German (see Limitations), for born-digital or manually corrected text, or for whole pages or long documents.

How to use

from transformers import pipeline

clf = pipeline("text-classification", model="Var3n/jobads-identification", tokenizer="stefan-it/zeitungs-lm-v1")
print(clf("Köchin für ein größeres Gasthaus wird gesucht.", truncation=True, max\_length=512))
print(clf("Eröffnung der neuen Eisenbahnlinie nach Villach.", truncation=True, max\_length=512))

 

Training data

The training data is based on the manually annotated ground truth created in the JobAds project (FWF P35783) and published with the CHR 2024 paper Who Advertises in Newspapers? Data Criticism in Mining Historical Job Ads (Venglarova et al., 2024). The ground truth is available at JobAds-FWFProject/Ground-Truth-CHR2024.

Source. The pages come from 29 newspaper titles of the ANNO corpus of the Austrian National Library, selected for having the largest number of editions (at least 10 years of issues each) between 1850 and 1950. The newspapers are predominantly in German. From pages preselected as containing job ads, one page per newspaper and year was randomly sampled, giving 3,300 pages. On these pages, all job advertisements were annotated manually with the tool doccano, resulting in 14,985 annotated job ads. Annotators marked the coordinates of each job advertisement on the page, so its position is known.

Construction of the dataset

1. The pages were OCRed with the frak2021 model (Mannheim University Library, 2021). The text regions were extracted from the resulting Page XML files.
2. The annotated coordinates were used to check whether an OCR text region overlaps with an annotated job advertisement.
3. Positive class: text regions that overlap with an annotated job advertisement by at least 75%.
4. Negative class: text regions with 0% overlap with any annotated job advertisement and with more than 10 characters.

Class	Examples
1 (job ad)	9,401
0 (other)	58,142

 

Class imbalance. The dataset is strongly imbalanced. We did not undersample the negative class, because that would strongly reduce the variability of the negative-class data (all other newspaper content). Instead, class weights were set to balanced during training, which gives more weight to the positive class.

Split. Exact duplicate texts and empty texts were removed. The data was then split randomly into 80% training (54,034 text regions) and 20% test data (13,509 text regions: 11,611 of class 0 and 1,898 of class 1), with a fixed random seed (42). The split is random at the level of text regions, not stratified, and not grouped by page or newspaper. Each text region is in exactly one of the two sets.

Training procedure

• Base model: zeitungs-lm-v1 (Schweter, 2024), an ELECTRA model pre-trained on historical German newspapers
• Task: binary sequence classification (ElectraForSequenceClassification with a two-class head)
• Loss: cross-entropy with class weights computed with scikit-learn's balanced mode on the training set (about 0.58 for class 0 and 3.60 for class 1)
• Tokenization: ElectraTokenizer of the base model, padding and truncation to a maximum length of 512 tokens
• Epochs: 5 (16,890 optimization steps)
• Batch size: 16 per device for training, 64 for evaluation
• Weight decay: 0.01
• Other settings: Hugging Face Trainer defaults (AdamW optimizer, learning rate 5e-5 with linear decay, seed 42, no mixed precision)
• Hardware and training time: about 3.6 hours on a single NVIDIA RTX A4000 GPU
• Model used: the best performing checkpoint
• Libraries: transformers 4.36.2, torch 2.13.0

Evaluation

All results are computed on the held-out test set (13,509 text regions). The AUC-PR is 0.969.

Classification report (predicted class = class with the highest probability):

Class	Precision	Recall	F1-score	Support
0 (other)	0.99	0.99	0.99	11,611
1 (job ad)	0.92	0.95	0.93	1,898
Accuracy			0.98	13,509
Macro avg	0.96	0.97	0.96	13,509
Weighted avg	0.98	0.98	0.98	13,509

 

Results broken down by newspaper or time period are not available.

Error analysis

We manually analysed 100 randomly drawn misclassifications. The errors fall into the following categories:

1. Short texts with ambiguous classification. Very short segments can be read either as a job ad or as advertising one's business. For example, "Maſſagen, Vodiekopa 18 9151" was annotated as a service offer, while "Eröffnung Maſſageſalon" was not annotated as an advertisement but predicted as one. This shows how hard it is to define a job advertisement, even for human annotators.
2. Texts cut by layout analysis errors. Errors in page segmentation produce fragments, e.g. only the contact part of an advertisement ("Schriftlichs Anfragen unter „Goloregen“ an die Annoncden- Exned. …"). Without the first part, such a fragment could belong to a job ad, a housing ad or any other advertisement.
3. Errors in the annotation. The labels were created in a single annotation round without a subsequent check, so some advertisements were overlooked. The model then predicts a positive class for a segment whose label is negative, although the prediction is correct. This lowers the measured performance.
4. Other types of advertisements. Advertisements share language patterns across categories. For example, a personal ad from the "Correspondence" section ("Intelligente Dame Mitle Dreißig, ſucht ehrb. Freundchaft mit keinfühlendem, helfendem herrn …") was falsely predicted as a job ad.
5. OCR errors in key vocabulary. The model is fairly robust to OCR errors in general, but fails when the most informative words are corrupted, such as Stelle ("position"), aufgenommen ("accepted"), Lehrling ("apprentice") or sucht ("looks for").
6. Advertisements in other languages. They are rare in the corpus, and the model sometimes, but not always, classifies them correctly. The model was not systematically evaluated on other languages than German.

Because many errors occur on genuinely ambiguous cases or are caused by annotation and segmentation problems, the model probably performs better in practice than the reported scores suggest.

Limitations and bias

• The model works on isolated text regions, so errors in the layout analysis (cut or merged regions) directly affect the predictions.
• Short and ambiguous segments (e.g. service offers) and other advertisement types (e.g. personal ads) are the main source of errors.
• Performance depends on the OCR quality. Corrupted key words lead to missed job ads.
• The model reflects the newspapers, regions and years in the training data. It may perform worse on other titles, layouts, typefaces or periods.
• Advertisements in languages other than German (e.g. French, English, Czech, Hungarian, Italian) are rare in the training data. Use with caution.

Citation

Ground-truth data

@inproceedings{venglarova2024advertises,
  title     = {Who Advertises in Newspapers? Data Criticism in Mining Historical Job Ads},
  author    = {Venglarova, Klara and Adam, Raven and M{\\"o}lzer, Wiltrud and Balasubramanian, Saranya and Kleinert, J{\\"o}rn and F{\\"u}llsack, Manfred and Vogeler, Georg},
  booktitle = {Proceedings of the Computational Humanities Research Conference (CHR 2024)},
  series    = {CEUR Workshop Proceedings},
  volume    = {3834},
  year      = {2024},
  address   = {Aarhus, Denmark},
  url       = {https://ceur-ws.org/Vol-3834/paper12.pdf}
}

 

There is no citation for this model yet. If you use it, please cite the ground-truth paper above and link to this repository.

Base model

@misc{schweter2024zeitungslm,
  author = {Schweter, Stefan},
  title  = {zeitungs-lm-v1},
  year   = {2024},
  url    = {https://huggingface.co/stefan-it/zeitungs-lm-v1}
}

 

OCR model: Mannheim University Library. (2021). Frak2021. https://ub-backup.bib.uni-mannheim.de/~stweil/tesstrain/frak2021/tessdata_best/frak2021-0.905.traineddata (accessed 20 November 2023).

Annotation tool: Nakayama, H. et al. (2018). Doccano: Text Annotation Tool for Human.https://github.com/doccano/doccano.

Acknowledgements

The ground truth was created in the project JobAds (FWF P35783), which was supported by the Austrian Science Fund (FWF). We thank the Austrian National Library for providing the data from the ANNO corpus. We thank our project colleague Wiltrud Mölzer and the project PI Jörn Kleinert for their support with ground-truth data annotation.

License

This model is released under the CC BY 4.0 license.

Authors and contact

University of Graz, Austria (JobAds project)

• Klara Venglarova (klara.venglarova@uni-graz.at) (model training)
• Raven Adam (raven.adam@uni-graz.at) (integrating model into a processing pipeline)
• Georg Vogeler (georg.vogeler@uni-graz.at) (supervision)