FootnotesExtractor/footnotes-extractor-models

FootnotesExtractor models

The model weights used by FootnotesExtractor, an open-source, interactive system that extracts and parses bibliographic references from the footnotes of scholarly books and articles.

Project page: https://footnotesextractor.github.io/FootnotesExtractor/

Install

You don't need to download anything by hand. The app's installer fetches these files into models/:

git clone https://github.com/FootnotesExtractor/FootnotesExtractor.git
cd FootnotesExtractor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python setup_models.py

The full instructions are in the README.

What is in this repository

The app runs these models one after another:

Step	File or folder	Architecture	Output
1. Find the footnote areas on a page	od/yolo11n_validated_v2.pt	YOLO11-n, 640 px	One class: footnote
2. Find the references in a footnote	ner/validated_v2_xlmr_nested/NER_reference/	XLM-R large	Reference
3. Tag the parts of a reference	ner/validated_v2_xlmr_nested/NER_component/	XLM-R large	17 labels, such as Author, BookTitle, PublicationYear and Ibidem
4. Split names	ner/validated_v2_xlmr_nested/NER_names/	XLM-R base	FirstName, LastName

Each folder holds a standard Hugging Face token-classification model with its tokenizer, plus label_map.json, which lists its labels. Steps 2 to 4 use BIO tags.

ner/validated_v2_nested/ holds a second set: XLM-R large for references, HerBERT large for parts and HerBERT base for names. python setup_models.py --herbert fetches it as well, and you choose it under Advanced Options when you upload.

Reading a page's text, splitting footnote areas into single footnotes and resolving ibid. and op. cit. are done by the app itself. Scanned pages are read with Surya OCR, whose weights are not part of this repository.

Limitations

• The models are trained on footnotes in open-access scholarly books. Handwriting, marginal notes and heavily degraded scans may give poor results.
• They make mistakes. The app's interactive viewer exists so that a reviewer can check and correct the output.

Licence

These weights are released under the GNU Affero General Public License v3.0, like the app.

They build on:

• the Ultralytics YOLO11 detector, AGPL-3.0
• xlm-roberta-large and xlm-roberta-base, MIT
• herbert-large-cased and herbert-base-cased by Allegro, CC BY 4.0 (second set only)

Commercial use: some of the books used for training are under non-commercial (-NC) licences. Commercial use of these models would require retraining without those books.

Authors and funding

• Michał Ulaniuk, National Information Processing Institute, Poland
• Emanuel Kulczycki, Humboldt-Universität zu Berlin; German Centre for Higher Education Research and Science Studies (DZHW)
• Przemysław Korytkowski, West Pomeranian University of Technology in Szczecin, Poland

Work co-financed by the state budget under the program of the Ministry of Education and Science called 'Science for Society II', project no. NdS-II/SP/0460/2023/01. We gratefully acknowledge Poland's high-performance computing infrastructure PLGrid and ACC Cyfronet AGH for providing computational facilities and support under computational grant no. PLG/2025/018513.