# From questions to insights: a reproducible question-answering pipeline for historiographical corpus exploration

Lucas Terriel, Vincent Jolivet

► To cite this version:

Lucas Terriel, Vincent Jolivet. From questions to insights: a reproducible question-answering pipeline for historiographical corpus exploration. Digital Humanities 2025: Accessibility & Citizenship, ADHO; NOVA FCSH University, Jul 2025, Lisbon, Portugal. hal-05171346

HAL Id: hal-05171346

https://hal.science/hal-05171346v1

Submitted on 19 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# From questions to insights: a reproducible question-answering pipeline for historiographical corpus exploration

Lucas Terriel

Vincent Jolivet

Session SP-46, DH2025, Lisbon, 18th Friday July 2025
Context (1): "Positions de thèses" corpus

ÉCOLE IMPERIALE DES CHARTES

POSITIONS DES THÈSES

SOLUTIONS PAR LES

ÉLÈVES DE LA PROMOTION 1866-67

POUR OBTEXIR

LE DIPLOME D'ARCHIVISTE-PALEOGRAPHE

PARIS

IMPRIMERIE SIMON RACON ET CIE

RUE D'ERFURTH, 1

1867

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

❖ École nationale des chartes (1821) : training heritage professionals and historians

❖ Position = longform abstract of dissertation (yearly publication since 1849 in the printed volume "Positions de thèses")
Context (2): “Positions de thèses” corpus

<figure><img src="image_4.png" /></figure>

❖ École nationale des chartes (1821) : training heritage professionals and historians

❖ Position = longform abstract of dissertation (yearly publication since 1849 in the printed volume “Positions de thèses”)

> 3000
longforms

± 5/10
pages

❖ Some of these works are pioneers in the renewal of analytical methods in history (French historians and philologists like Michel Pastoureau, Olivier Guyotjeannin, Paul Meyer etc.)

❖ Corpus is online: https://theses chartes psl eu/

❖ Alls texts in XML-TEI available via DTS API (Distributed Text Service) endpoint
Context (3): "Positions de thèses" corpus - limitations

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Navigation challenge

Limited search capabilities = filters + full-text search
=> noisy results

❖ École nationale des chartes (1821) : training heritage professionals and historians

❖ **Position = longform abstract of dissertation** (yearly publication since 1849 in the printed volume "Positions de thèses")

> 3000
longforms

± 5/10
pages

❖ Some of its works are pioneers in the renewal of **analytical methods in history** (French historians and philologists like Michel Pastoureau, Olivier Guyotjeannin, Paul Meyer etc.)

❖ Corpora is online: https://theses chartes psl eu/

❖ Alls texts in XML-TEI available via DTS API (Distributed Text Service) endpoint
# Goals - Why a RAG for “Positions de thèses”?

* **Similar documents... according to what?** Based on previous work presented at DH2022 (Jolivet & Torres Aguilar, 2022): semantic models (Doc2Vec, DistilBERT) + lexical clues (keywords with KeyBERT, named entities) → identify related thesis abstracts.

* In context of application: a **complementary RAG-based (LLM framework) approach**:

* **(1)** Provide an domain-specific

* **(2)** Improve the website experience

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

**assistant** to answer natural language questions

<figure><img src="image_9.png" /></figure>

## Benefits for historians

- Build thematic collections

- Compare historiographical perspectives on events or figures

- Track methodological trends

- Run quantitative explorations

Like a **librarian or archivist**, RAG guides users through complex and vast collection of books or archives

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

BY
## Credits

* Vincent Jolivet

* MPN Team (Philippe Pons, Jean-Victor Boby)

* Frédérick Madore (Leibniz-Zentrum Moderner Orient researcher)

* Léonard de Bengy (internship work on prompting)

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
# RAG Pipeline (1) - Data processing (chunking) & indexing step

two-step chunking strategy (section + token-based)

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>
# RAG Pipeline (2) - Retriever step

two-step chunking strategy (section + token-based)

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>
# RAG Pipeline (3) - Reader step (LLM generation)

two-step chunking strategy (section + token-based)

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

## PRE-PRODUCTION (Extract, transform & load ops)

Vectorization

## PRODUCTION (Retrieve & generate)
# RAG Pipeline (4) - Tools and demo interface

Tech stack: run locally (on lightweight hardware using GPU or not) + well-documented and open-source tools

<figure><img src="image_28.png" /></figure>

LM Studio

<figure><img src="image_29.png" /></figure>

LLM server

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

LanceDB

<figure><img src="image_33.png" /></figure>

LangChain

sentence-transformers/

distiluse-base-multilingual-cased-v2

Lajavaness/sentence-camembert-base

Lajavaness/sentence-camembert-large

RAG Framework

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

Streamlit

User interface

<figure><img src="image_36.png" /></figure>

Streamlit application to test RAG with different configurations
→ **experiment quickly**, but... *"It seems to work"* is not enough in research context.

**A reproducible evaluation framework**, focusing on two components:

* **Retriever**: Does it retrieve the relevant documents?

* **Reader**: Does it generate **useful, source-grounded** answers?
Evaluation framework (2) - Retriever

<figure><img src="image_37.png" /></figure>

❖ Classic **Information Retrieval (IR) task**.

❖ Task: Check if the retriever finds **relevant documents**

**Method:**

❖ Build **synthetic Q-A-context triplets** with LLM on our thesis corpus

❖ Evaluate key parameters: embedding models, chunk sizes and impact of optimizations (hybrid search, reranking)

❖ IR Metrics: Use **Recall@k**, **Exact Match**

QA - context synthetic dataset (JSON)
# Evaluation framework (3) - Reader

* **Harder problem**

* Classical ML evaluation failed
  → no “correct” answer target

Focus on **judgment-based criteria** (Dong et al., 2024):

* Usefulness – helpful to a researcher

* Groundedness – faithful to sources

* Standalone quality – understandable alone

LLM-as-a-judge method:

* Evaluation prompt = {question + context + reference answer + generated answer}

* Scoring on 3 criteria (between 1-5)

<figure><img src="image_38.png" /></figure>

**ComparA** interface: 2 llm “judge” our RAG’s answer based on evaluation prompt (question, context and target response), the system returns feedback and a score (1-5)
# Evaluation framework (4) - Limitations, biases

* LLM judges other LLM!

* auto-validation bias

* no human expert judgment

## What we observed on question (using Streamlit application):

* **Correct on:** factual, historical, methodological

e.g. "How were diplomatic relations conducted by King Charles VII?"

* **Issues with:** comparative, quantitative, prospective

e.g. "What were the dominant historiographical medieval themes from between 1970 and 1980?"

## Next directions:

* Synthetic dataset → needs **expert refinement** - **expert-in-the-loop** process

* Metadata filtering in vector store → **more precise retrieval** for historical data

* API reasoning → LLM has a **direct access to structured** data like JSON

<figure><img src="image_39.png" /></figure>
ROADMAP

<figure><img src="image_40.png" /></figure>

Expanding the knowledge base

<figure><img src="image_41.png" /></figure>

Retriever based on Metadata integration and filters

<figure><img src="image_42.png" /></figure>

User acceptability and feedback

<figure><img src="image_43.png" /></figure>

Security and robustness (ML-based or rule-based guardrails)

<figure><img src="image_44.png" /></figure>

Evaluation criteria / expert validation

<figure><img src="image_45.png" /></figure>

Scalability & deployment

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

Thank you

<figure><img src="image_49.png" /></figure>
Bibliography

Camps, Jean-Baptiste / Baumard, Nicolas / Langlais, Pierre-Carl / Morin, Olivier / Clérice, Thibault / Norindr, Jean (2023): "Make Love or War? Monitoring the Thematic Evolution of Medieval French Narratives", in: Computational Humanities Research (CHR 2023). HAL: 04250657.

Devlin, Jacob / Chang, Ming-Wei / Lee, Kenton / Toutanova, Kristina (2019): "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", arXiv:1810.04805.

Dong, Liming / Lu, Qinghua / Zhu, Liming (2025): A Taxonomy of AgentOps for Enabling Observability of Foundation Model based Agents. arXiv:411.05285v1

Douze, Matthijs / Guzhva, Alexandr / Deng, Chen / Johnson, Jeff / Szilvasy, Gergely / Mazaré, Pierre-Emmanuel / Lomeli, Manuel / Hosseini, Laleh / Jégou, Hervé (2024): "The Faiss library", arXiv:2401.08281.

Guyotjeannin, Olivier (1997): "Naissance de la thèse (1849-1914)", in: L'École nationale des chartes, histoire de l'école depuis 1821. Paris: École nationale des chartes.

Jolivet, Vincent / Torres Aguilar, Sergio (2022): "Document similarity and topic clues. A historiographical study case", in: Digital Humanities 2022 Conference, Tokyo (Japan). HAL: 04087517.

Le, Quoc V. / Mikolov, Tomas (2014): "Distributed Representations of Sentences and Documents", arXiv:1405.4053.

Martin, Louis / Muller, Benjamin / Suárez, Pedro Javier Ortiz / Dupont, Yoann / Romary, Laurent / de La Clergerie, Éric Villemonte / Seddah, Djamé / Sagot, Benoît (2020): "CamembERT: a Tasty French Language Model", in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020). arXiv:1911.03894.

Mikolov, Tomas / Chen, Kai / Corrado, Greg / Dean, Jeffrey (2013): "Efficient Estimation of Word Representations in Vector Space", arXiv:1301.3781.

Reimers, Nils / Gurevych, Iryna (2019): "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", in: Conference on Empirical Methods in Natural Language Processing 2019 (EMNLP 2019). arXiv:1908.10084.

Vaswani, Ashish / Shazeer, Noam / Parmar, Niki / Uszkoreit, Jakob / Jones, Llion / Gomez, Aidan N. / Kaiser, Łukasz / Polosukhin, Illia (2023): "Attention Is All You Need", arXiv:1706.03762.