# Pandore: automating text-processing workflows for humanities researchers

Floriane Chiffoleau, Mikhail Biriuchinskii, Glenn Roe, Motasem Alrahabi

<figure><img src="image_1.png" /></figure>

## To cite this version:

Floriane Chiffoleau, Mikhail Biriuchinskii, Glenn Roe, Motasem Alrahabi. Pandore: automating text-processing workflows for humanities researchers. DH2025 - Accessibility & Citizenship, Jul 2025, Lisbon, Portugal. hal-04986730

HAL Id: hal-04986730

https://hal.science/hal-04986730v1

Submitted on 11 Mar 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Pandore: automating text-processing workflows for humanities researchers

Observatoire des textes, des idées et des corpus (ObTIC) - Sorbonne Université

Floriane Chiffoleau, Mikhail Biriuchinskii, Glenn Roe, Motasem Alrahabi

prenom.nom@sorbonne-universite.fr

## Introduction

The recent integration of AI-powered workflows in digital humanities projects has led to the development of new tools and software to (semi-)automatically work on textual editions. However, many of these tools are less than intuitive and can be hard to grasp for less technically-savvy users. To address this issue, several data-science pipelines have been created to facilitate access to large digitized collections, including EHRI [Bénière, Chiffoleau, and Scheithauer 2024] and Gallic(orpor)a [Sagot, Benoît et al. 2022], among others. Likewise, Pandore<fnref n="1" /> aims to provide user-friendly access to a suite of digital tools to collect, prepare, convert, and analyze data, extract relevant insights, and create interpretive visualizations.

In this presentation, I will introduce the Pandore interface as it was before 2025, before expanding on the recent implementations based on user feedback, reflecting on its ongoing development in the context with our partnership with the French national library (BNF)<fnref n="2" /> and their DataLab<fnref n="3" />, and finally concluding with the current limits we are encountering.

## Description of the toolbox

Pandore was developed by our team at Sorbonne Université, ObTIC, the Observatory for Texts, Ideas and Corpus<fnref n="4" />, dedicated to the design and experimentation of new digital methodologies for humanities researchers, under the supervision of Motasem Alrahabi<fnref n="5" /> and Glenn Roe<fnref n="6" />. The first version was made by engineers, PhD candidates and postdoctoral researchers [Cordova et al., 2022; Alrahabi et al., 2023], and the latest modifications and improvements have been implemented this year by my colleague, Mikhail Biriuchinskii<fnref n="7" />, and me<fnref n="8" />.

The Pandore toolbox integrates numerous digital tools to assist users in automating various tasks aimed at creating and exploiting textual corpora, and it also allows users to combine multiple tasks in the form of automated workflows or pipelines. Among the functionalities present in the toolbox, many offered multiple tools for a given task, meaning that users can

<footnote n="1">¹ https://obtic-gpu1.mesu.sorbonne-universite.fr:8550/</footnote>

<footnote n="2">² https://www.bnf.fr/fr</footnote>

<footnote n="3">³ https://www.bnf.fr/fr/bnf-datalab</footnote>

<footnote n="4">⁴ https://obtic.sorbonne-universite.fr/</footnote>

<footnote n="5">⁵ https://obtic.sorbonne-universite.fr/equipe-permanente/motasem-alrahabi/</footnote>

<footnote n="6">⁶ https://obtic.sorbonne-universite.fr/equipe-permanente/glenn-roe/</footnote>

<footnote n="7">⁷ https://obtic.sorbonne-universite.fr/equipe-permanente/mikhail/</footnote>

<footnote n="8">⁸ https://obtic.sorbonne-universite.fr/equipe-permanente/floriane-chiffoleau/</footnote>
test different outputs and pick the one that best suits their research question, before finally executing the task.

Seven essential tasks were initially proposed: optical character recognition or OCR, format conversion, automatic annotation tasks, data mining, visualization, corpus collection, text correction, and pipelines. Format conversion allows users to either encode a text in TEI XML, or more generally, to convert digitized documents from one format to another, notably with the tool Teinte<fnref n="9" />. The automatic annotation module retrieves part-of-speech tagging, semantic categories and named entities recognition or NER, while the data mining module performs various information extraction tasks, including keywords extraction, using KeyBERT [Sharma et Li, 2019], or topic modelling. In terms of visualization, users can generate geolocated maps, networks of characters or co-occurrences, as well as corpus statistics and ontological relationships in an annotated corpus using Ariane<fnref n="10" /> [Alrahabi 2021], another tool developed by ObTIC. Pandore also enables users to create their own corpora of digitized texts using available APIs, either from Wikisource<fnref n="11" /> or Gallica<fnref n="12" />. Finally, with pipelines, it is possible to combine OCR and NER, to retrieve the named entities of an image that has been transcribed; or do the extraction of spatial or topographical named entities from a text (transcribed or not), which are then geolocated on a digital map [Koudoro-Parfait, Alrahabi, Dupont, Lejeune, & Roe 2023]. Among the functionalities present in the toolbox, many offer multiple tools for a given task, meaning that users can test different outputs and pick the one that best suits their research question, before finally executing the task. In the case of ATR, for instance, users can choose to use Tesseract<fnref n="13" /> (printed texts), or our EScriptorium instance<fnref n="14" /> (handwritten or typescript documents). Likewise, for NER tasks, users can select three different tools: spaCy<fnref n="15" />, Flair<fnref n="16" /> or BERT<fnref n="17" />. Furthermore, these NER tools can be used to treat texts in multiple languages, including English or Spanish, as well as additional language models that were added recently.

## Incorporating user feedback, improving the user experience and expanding the toolbox

Building upon several campaigns of targeted user-feedback, the beta version of Pandore was significantly upgraded in order to address user-identified bugs, and to improve its overall design, documentation, and user interface.

Several modifications were done on the Pandore toolbox. First, the design of Pandore itself was modified (Figure 1). The color palette was changed to render the instance less austere and more cheerful, and many hyperlinks were added in order to provide a smoother and more seamless navigation between modules. We also added a set of new tutorials<fnref n="18" /> and

<footnote n="9">⁹ https://obtic.huma-num.fr/teinte/</footnote>

<footnote n="10">¹⁰ https://obtic.huma-num.fr/ariane/</footnote>

<footnote n="11">¹¹ https://fr.wikisource.org/wiki/Wikisource:Accueil</footnote>

<footnote n="12">¹² https://gallica.bnf.fr/</footnote>

<footnote n="13">¹³ https://github.com/tesseract-ocr/tesseract</footnote>

<footnote n="14">¹⁴ https://obtic-gpu1.mesu.sorbonne-universite.fr:8501/</footnote>

<footnote n="15">¹⁵ https://spacy.io/</footnote>

<footnote n="16">¹⁶ https://huggingface.co/flair</footnote>

<footnote n="17">¹⁷ https://huggingface.co/google-bert</footnote>

<footnote n="18">¹⁸ https://obtic-gpu1.mesu.sorbonne-universite.fr:8550/tutorial</footnote>
documentation<fnref n="19" /> where needed, and modified existing ones, all of which will allow users to properly and efficiently employ each tool and workflow. Lastly, the instance has been migrated to a GPU for a faster and more efficient use.

<figure><img src="image_3.png" /><figcaption>Figure 1: Interface of the Pandore toolbox</figcaption></figure>

Other modifications were made, directly to the existing tasks, with two goals: switching to another library for better use and/or adding more options. Therefore, part-of-speech tagging and error correction switched, respectively, from SEM<fnref n="20" /> and Jamspell<fnref n="21" />, to spaCy and LanguageTool<fnref n="22" />, which also added many new languages to their task. Three new corpus options, Project Gutenberg<fnref n="23" />, Google Books<fnref n="24" /> and URLs, have been added to the “Corpus collection” task. The visualization tool Tanagra now offers two versions, both map places<fnref n="25" />, but one also provides sentiment analysis<fnref n="26" />. Finally, a new version is proposed for the TEI XML conversion tool<fnref n="27" />, as it is now possible to add more metadata options, to implement options for the body of text, all with a modified and more convenient HTML template.

Regarding the implementation of future functionalities, in addition to tasks such as automatic speech recognition (ASR) and quotation extraction, our objective was to integrate more tasks oriented towards natural language processing (NLP) workflows, as some can be very useful for humanities researchers but are not always easy to grasp. Through the use of interoperable and modular Python scripts, these NLP pipelines would encompass tasks such as preprocessing and normalization (lemmatization), statistical language analysis (word

<footnote n="19">¹⁹ https://obtic-gpu1.mesu.sorbonne-universite.fr:8550/documentation</footnote>

<footnote n="20">²⁰ https://github.com/YoannDupont/SEM</footnote>

<footnote n="21">²¹ https://github.com/bakwc/JamSpell</footnote>

<footnote n="22">²² https://languagetool.org</footnote>

<footnote n="23">²³ https://www.gutenberg.org/</footnote>

<footnote n="24">²⁴ https://books.google.fr/</footnote>

<footnote n="25">²⁵ https://obtic.sorbonne-universite.fr/tanagra/map</footnote>

<footnote n="26">²⁶ https://github.com/obtic-sorbonne/Tanagra2/blob/main/Tanagra.ipynb</footnote>

<footnote n="27">²⁷ https://obtic-gpu1.mesu.sorbonne-universite.fr:8550/conversion_xml</footnote>
frequency), advanced text analysis (readability scoring) and even text generation (translation).

## Shaping the relations between the team and the BnF

Thanks to an ongoing partnership between Sorbonne Université and the BNF, the ObTIC team is currently in residence at the BNF DataLab, a service intended for researchers who wish to work on the BNF's considerable digital resources. One common shared objective between our two teams is to help humanities researchers with the automatization of their workflows through targeted access to BNF data. As such, the Pandore toolbox has progressively integrated the Gallica digital library ecosystem. We are thus actively developing several Gallica-related tools, including corpus extraction via the Gallica API, and Gallica-Obvie<fnref n="28" />, a tool that combines the advanced text-search capabilities of Obvie<fnref n="29" />, including lexical statistics and text comparison [Alrahabi 2022], with the BNF's digital collections.

## Remaining limits and issues to the Pandore instance

Three current issues remain, despite the instance' upgrades. There are still some server access problem, with issues of web availability and access rights. The instance also contains outside applications, which are hard-to-implement directly to the instance and then require to locally install it, such as Minerva<fnref n="30" />, Textolab<fnref n="31" /> or Summarization<fnref n="32" />. Lastly, despite the installation on the GPU, some time out problems are still unresolved, notably about size data.

Despite those remaining inconveniences, we are confident that the instance can provide much help to humanities researchers and be a true asset to ease their work.

<footnote n="28">²⁸ https://obtic.huma-num.fr/gallica-obvie/</footnote>

<footnote n="29">²⁹ https://obtic.huma-num.fr/obvie/</footnote>

<footnote n="30">³⁰ https://github.com/YoannDupont/minerva</footnote>

<footnote n="31">³¹ https://github.com/obtic-sorbonne/Textolab</footnote>

<footnote n="32">³² https://github.com/obtic-sorbonne/Summarization</footnote>
## Bibliography

Alrahabi, M. (2021, Jan.). Ariane: dispositif de fouille et de lecture synthétique de textes. *DigitAl Humanities and cultural herItAge: data and knowledge management and analysis (Atelier Dahlia)*, Montpellier (virtuel), France. [hal-03167271](https://halshirpaHAL-03167271)

Alrahabi, M. Obvie: interface web pour la fouille et la comparaison de textes. *Atelier DigitAl Humanities and cultural herItAge: data and knowledge management and analysis durant la conférence francophone sur l'Extraction et la Gestion des Connaissances (egc2022)*, Jan 2022, Blois, France. [hal-03543362](https://halshirpaHAL-03543362)

Alrahabi, M., Fedchenko, V., Petkovic, L., Roe, G., Cordova, J. (2023, July 10-14). *Pandore: a toolbox for digital humanities text-based workflows*. Poster. Digital Humanities 2023. Collaboration as Opportunity (DH2023), Graz, Austria.

Bénière, S., Chiffoleau, F., & Scheithauer, H. (2024, June). Streamlining the Creation of Holocaust-related Digital Editions with Automatic Tools. In *EHRI Academic Conference - Researching the Holocaust in the Digital Age*, EHRI-3, Warsaw, Poland. [url](https://ehri.org/conferences/ehri3)

Cordova, J., Dupont, Y., Petkovic, L., Gawley, J., Alrahabi, M., Roe, G. « Toolbox : une chaîne de traitement de corpus pour les humanités numériques » in TALN 2022 (Démo), Paris, France, [url](https://www.taln.org/lms/abstract/1007)

Koudoro-Parfait, C., Alrahabi, M., Dupont, Y., Lejeune, G., & Roe, G. (2023, June 30). Mapping spatial named entities from noisy OCR output: Epimetheus from OCR to map. *Digital Humanities 2023. Collaboration as Opportunity (DH2023)*, Graz, Austria. [doi](https://doi.org/10.5407/dh2023.1007)

Sagot, B., et al. (2022, Dec.). Gallic(orpor)a : Extraction, annotation et diffusion de l'information textuelle et visuelle en diachronie longue. In *DataLab de la BnF : Restitution des travaux 2022*, DataLab de la BnF, Paris, France. [url](https://datalab.bnf.fr/lms/data/10.5407/dh2022.1007)

Sharma, P., & Li, Y. (2019). Self-Supervised Contextual Keyword and Keyphrase Retrieval with Self-Labelling. *Preprints*, 2019080073. [doi](https://doi.org/10.2019./2019080073)